Putting AI Inside Climate Models Makes Them Harder to Check, Modelers Argue

Take a climate model that runs on one supercomputer, move it to another, and you are not guaranteed to get the same climate back. Massonnet and colleagues, writing in Geoscientific Model Development in 2020, built a protocol to test exactly that on the EC-Earth3 model. They found both outcomes: an older version whose results no longer matched once the machine underneath it changed, most likely because of the way it handled river runoff along the Antarctic coast, and a newer version that came through the move intact. Their advice was to assume the worst. The default, they wrote, should be that an Earth system model is not replicable when the computing environment changes, until someone has proven that it is.

That was a model made entirely of equations people had written down. The ones now arriving have learned components inside them, trained on data rather than derived from theory. On Aug. 28, Nature Machine Intelligence published a Perspective arguing that this changes what reproducibility means for such models. Its 23 authors are co-led by Min Chen and Zhiyi Zhu of Nanjing Normal University, with co-authors including Thorsten Wagener, Niklas Boers and Gustau Camps-Valls. Artificial-intelligence-infused hybrid Earth system models, they write, "face amplified issues of numerical instability, procedural opacity and asymmetric access to computational resources." As a Perspective, the paper presents a diagnosis of the field rather than new experimental evidence, identifying numerical instability, procedural opacity and unequal access to computational resources as key reproducibility challenges.

Start with the first. Bitwise reproducibility is fragile even without AI in the model. Beate Geyer, Thomas Ludwig and Hans von Storch ran a regional atmosphere model called COSMO-CLM5.0 across different computer platforms. Identical numerical results are possible, they reported in 2021, only if everything is fixed — the computer, the compiler, the chosen options, the boundary values and the initial conditions — and if the order of mathematical operations does not change between runs. Otherwise, they concluded, the most anyone should expect is that two runs come out statistically identical. Adding a trained component adds to the list of things that must be held fixed, starting with the weights that came out of one particular training run.

"Procedural opacity" is the second fault. A hand-written piece of model physics can be read: the equations are in the paper, and another group can build it again from the description. A learned component does not work that way. Its behavior sits in numbers produced by a particular training set, a particular training procedure and a decision about when to stop, so an account of what the model actually did has to include all three. Whether they travel with the model is a matter of practice, not of physics.

The third phrase, "asymmetric access to computational resources," is the plainest of the three. Checking a simulation means running it again, and running it means having a machine that can. The protocol Massonnet's team built exists because groups run the same model on different systems in the first place. The consortium behind that model, 27 institutions in Europe, adopted the protocol as a standard check on any new model version, including for CMIP6, the coordinated round of runs climate-modeling groups carry out together.

What the authors say could follow, they keep conditional. If these challenges are left unaddressed, they write, hybrid models "risk turning" into "opaque and weakly verifiable systems," reducing traceability, weakening cumulative knowledge building and "narrowing the evidential basis for climate risk assessment and policy guidance." The claim is about where the practice could end up, not where it is.

The remedy they offer is a proposal, not a tool anyone can pick up today. They call for reproducibility to be reframed for hybrid models, and propose what they describe as "an integrated roadmap that couples a theory of reproducibility assessment with practical pathways for implementation in modelling practices." Inside it sits a reference guideline they name Reproducibility in hybrid Earth system models, or RHEM, introduced "for governing transparent, trustworthy, and reproducible hybrid ESMs." Their closing argument is that reproducibility should be "repositioned from a post hoc requirement to a structural property" of these models, designed in from the start rather than certified afterward. What RHEM asks of a modeling group is not public.

Building on these earlier studies, the Perspective examines how machine learning can amplify reproducibility challenges in hybrid Earth system models and proposes a shared framework for addressing them. The tests that exist today were designed for models built out of equations. Whether they carry over to models with learned parts inside them is the question these 23 authors are putting to their field.