What Makes an AI Landslide Map Fail Is the Rock, Not the Algorithm
Twelve geologists, six months, a thousand square kilometers of hillside. That was the price of mapping by hand what two rainstorms did to Emilia-Romagna in May 2023. Tens of thousands of slopes gave way inside a fortnight, and every scar had to be traced from aerial photographs while civil protection crews waited to learn which valleys to reach first.
Nicola Dal Seno and Matteo Berti of the University of Bologna, with Alessandro Corsini of the University of Modena and colleagues, wanted to know whether a machine could have done that first pass. Their answer, published Thursday in the peer-reviewed journal Natural Hazards and Earth System Sciences, is a qualified yes with a geological condition attached.
The test was built to be unfair in the way a real emergency is unfair. Both networks were trained on one municipality only: Casola Valsenio, about a tenth of the ground the storms had hit. U-Net is a convolutional workhorse borrowed from medical imaging; SegFormer is a newer transformer design. Neither was retrained before being applied to three neighboring municipalities it had never seen, and both were scored against the geologists' own hand-drawn inventory.
A ladder of input combinations was tried, from free Sentinel-2 satellite imagery alone up to a stack that added purpose-flown aerial photography, a vegetation-change map, and a slope model. On its home ground, U-Net's overlap with the hand map climbed from 0.56 on the leanest input to 0.73 on the richest, an F1 score, where a perfect match would be one. SegFormer's range was narrower, and it cared less what it was fed. Which architecture you choose barely mattered; the paper's summary is that data quality, not architecture, drives performance.
The result with the most immediate practical weight is not the best score. It is the second-cheapest configuration, which uses nothing but a post-event Sentinel-2 image and a vegetation-change map computed against a Sentinel-2 scene from the year before: data that is free, global, and usually available within days. In the three municipalities the models had never seen, that configuration came within 0.06 of the richest setup, and often closer. The aerial flights bought a little accuracy, not a different class of map.
On those three, the best scores landed between 0.60 and 0.63: lower than at home, but still maps a geologist could work from. The errors were consistent and explainable. Freshly plowed fields around Modigliana registered as landslides, because bare tilled soil reflects light as much as a fresh scar does. Shaded slopes in Predappio went unmapped, there being too little light coming back to read. Flood-scoured riverbanks near Brisighella were flagged as failures that had not happened.
Those numbers look poor next to the scores machine-learning papers usually report, and the paper spends a paragraph explaining why they are not. Landslide pixels are a tiny fraction of any of these images, so a model that guessed "no landslide" everywhere would still be right almost all the time. Scores in this range are ordinary in the landslide-mapping literature.
Northern Brisighella is not flysch. It is Pliocene Blue Clay, and clay fails differently: earth flows, and earth slides with lobate toes, in badland country that is pale and thinly vegetated before anything moves at all. The models had never been shown a landslide of that kind. Their best score there was 0.53, and their worst 0.29, and that is not a map anyone should plan from. The authors are careful about the claim: the poor generalization is "associated with" the dominance of Blue Clay and the narrow lithological range of the training data, not shown to be caused by it.
Two follow-up tests make the association hard to wave away. Feeding the network a lithology map as an extra input, the obvious fix, made things worse: on Blue Clay ground the model then found almost nothing at all, an F1 of 0.02, having evidently learned to associate landslides with the one rock type it was trained on. Retraining the same configuration on Blue Clay and testing it there raised the score from 0.32 to 0.41. Better, and still not good.
The task the Emilia-Romagna emergency commission needed done first was not a map but a list: which buildings stood inside a landslide, or close enough to its edge to be at risk. On that question, the four best configurations found, on average, 528 of the 654 structures the manual survey had flagged. It is a different measurement from mapping accuracy, with its own misses and false alarms, and both kinds cost something: a house wrongly cleared and a household wrongly moved out are both expensive mistakes.
None of this makes the models better mappers than the geologists, and the arithmetic of the study rules it out. The hand-drawn inventory is the yardstick every score is measured against, so no model can out-score it. What a model can do is produce a first draft in hours rather than months, which the authors argue could save weeks of digitizing in a crisis. The paper's abstract keeps the boundary where it belongs: manual revision remains necessary.
The evaluation has a family resemblance the authors disclose. The reference inventory is their own team's, and the three experts who ranked the output maps for readability are three of the paper's authors, although they scored them blind to which configuration had produced which map. The inventory is public under a Creative Commons license; the model code is available from the authors on request. For a hazard office elsewhere, the instruction that comes out of all this is narrow and checkable: before trusting a map like this over unfamiliar ground, find out whether the rock beneath it was in the training set.