Better Forecasts, or Better-Value Ones? What Happens When You Put a Price on an AI Weather Model

A drainage crew has until Thursday evening to decide: put the flood barriers out, or leave them in the depot. The forecast offers a probability of heavy rain, not a promise of it. Deploying costs a known amount in overtime, closed roads and public patience. Leaving the barriers where they are costs nothing at all, right up until the water arrives. Nobody in that room asks which model posted the lower root-mean-square error last quarter, and none of the scores that forecasting centres publish answers the question the crew actually has.

Closing that gap is the point of a paper by Leonardo Olivetti and Gabriele Messori of Uppsala University, working with Paolo Avner and Stephane Hallegatte of the World Bank's Global Facility for Disaster Reduction and Recovery. Their framework prices a forecast by the decisions it drives rather than by how closely it matched the weather, and they use it to compare two systems from the European Centre for Medium-Range Weather Forecasts: the physics-based IFS HRES and the data-driven AIFS. Environmental Research Letters published the peer-reviewed study on July 31.

That date is misleading on its own. The same four authors presented the work at the EGU General Assembly in Vienna on May 3-8, under the title "From Forecast Skill to Forecast Value: Do AI Weather Forecasts Deliver Real-World Economic Benefits?" It appeared again on June 2 as World Bank Policy Research Working Paper 11407, whose abstract reads almost sentence for sentence like the journal version's. What is new at the end of July is peer-reviewed publication, not the finding. The World Bank and the GFDRR funded the work, and two of the four authors are World Bank staff, so its framing is the one a development lender brings: what does this forecast buy, and for whom.

Cost, loss, and the ratio between them

Decision analysts have a name for the drainage crew's problem. It is the cost-loss ratio: the price of protecting yourself, divided by the loss that protection spares you. When protection is cheap next to the damage, it pays to act on weak signals and to swallow a great many false alarms. When protection is expensive, it pays to wait for a strong signal and accept that some events will land on you unprotected. Hand the identical forecast to two users sitting at different ratios and it is worth different amounts to each. A city that can sandbag a metro entrance for a few thousand euros is not making the same decision as a port that has to stop loading.

The textbook version of this calculation treats each event as though it stood by itself. Extremes seldom oblige. Olivetti and colleagues add two penalty functions to the standard setup. One accounts for compounding losses across multiple extreme events, so that a storm landing on a city still mopping up from the last one costs more than the same storm landing on a city at full strength. The other accounts for declining user trust after repeated false alarms, on the reasoning that a warning nobody believes any more is worth less than its hit rate suggests. Note what that second penalty is: a modelling choice inside the framework. The study does not measure how any real population responds to being warned and then spared.

Where the ranking flips

Applied to cities exposed to extreme weather hazards, the comparison declines to name a winner. It produces a crossover instead. In the authors' own words, in some cities in Southern Europe "the higher sensitivity of the physics-based model IFS HRES makes it better-suited when protection costs are small relative to potential losses, while the higher specificity of the data-driven AIFS makes it better when protection costs are higher."

Sensitivity and specificity are doing all the work in that sentence, and they are worth unpacking. A sensitive system raises the alarm readily. It catches more of the events that matter, and it cries wolf more often. A specific system fires less often and is right more of the time when it does fire. Neither is inherently better. Which one you want turns entirely on whether a false alarm or a missed event is the more expensive mistake in your particular situation, which is another way of saying it turns on your cost-loss ratio.

There is an independent reason to expect that asymmetry between the two kinds of model. Data-driven systems are trained to minimise average error, and the cheapest route to a low average error is to hedge toward the middle of the distribution, smoothing away the tails where extremes live. In May, a separate group at ETH Zurich, the Karlsruhe Institute of Technology, the Helmholtz Centre for Environmental Research and the University of Geneva reported in Science Advances that physics-based models outperform AI forecasts of record-breaking extremes, with the AI systems consistently underestimating both the frequency and the intensity of records. Different team, different method, same direction of travel.

That is why the question is not academic. ECMWF runs both kinds of system, and any agency consuming that output has to decide not which model is more skilful in general, but which one to wire into an early-warning chain that triggers real spending by real municipalities.

What the paper does not say

The result is narrower than a headline would like it to be. The crossover is illustrated for some cities in Southern Europe; the abstract names none of them and gives no count. This is a framework demonstrated on case cities, not a league table of forecasting systems. The authors' own summary points away from any verdict on AI at all: the value of forecasts, they write, is "highly sensitive to assumptions about compounding losses, penalty structures, and prevention costs," often "substantially altering conclusions drawn from meteorological skill alone." That sensitivity is the finding. Compressed into "physics beats AI," or into its mirror image, it becomes a claim the authors wrote the paper specifically to head off.

Before anyone can say which model is worth more to a given city, that city has to put a figure on a day of unnecessary precaution, and a figure on the flood it failed to prepare for. Those are municipal numbers, not meteorological ones. The framework needs both of them before it will return an answer.