Why Two Nearly Identical AI Training Runs Produce Different Models

Fix every random seed. Pin the order the training data arrives in. Run it on a processor whose arithmetic is repeatable. Now change one number in the starting network by a single bit, the smallest difference a computer can hold, and train again. You can get a different model: different weights, and different answers on a small but stubborn fraction of your test inputs. Machine learning engineers have known this for years and have mostly filed it under nuisance.

A paper published Nov. 5, 2026, in Nature Machine Intelligence argues that this is not a nuisance but a fact about the shape of the problem. Andrew Ly and Pulin Gong of the School of Physics at the University of Sydney report a limit on reproducibility that does not loosen as precision improves, and they borrow their explanation from the study of chaotic physical systems in the 1990s: riddled basins.

A basin is the whole set of starting points that end at one outcome. A riddled basin is shot through with points that end somewhere else, at every magnification. The paper puts it this way: "every initialization leading to one solution lies arbitrarily close to initializations leading to different solutions, forming a fat fractal with structure at arbitrarily fine scales." Magnify the map and the braiding never resolves.

That separates it from a result the field already has. In 2024, Jascha Sohl-Dickstein showed, in a preprint, that the boundary between the settings chosen before training that let a network learn and the ones that leave it stuck is fractal. Ly and Gong call boundaries of that kind "skinny" fractals, occupying no volume: a starting point far enough from one sits in clear ground, and more precision buys safety. They argue their own maps show a fat fractal instead, which takes up volume and leaves no clear ground to sit in, so that "no increase in precision offers recourse."

The proof lives in a network with four weights

Riddling is not a loose analogy. Dynamical systems theory says what is sufficient to produce it: a chaotic state inside a region training cannot leave, a vanishingly thin set of points in it from which tiny sideways nudges grow instead of dying, and at least one rival destination elsewhere. Checking all that inside a real network is out of reach, and the authors say so: "Because direct stability analysis of high-dimensional attractors in deep network training is analytically and computationally intractable, we adopt a minimal modelling approach."

So the proof is done in a toy. Their minimal model is a two-layer network with four weights, trained by ordinary gradient descent to fit a handful of random points, the smallest network with a symmetry worth having: swap its two hidden units and the function is unchanged. All of the conditions hold there. Then they measure the quantity that decides the practical question: the uncertainty exponent says how fast the chance of landing somewhere different falls away as the starting point is tightened. In the toy, it comes out at 0.0126, near zero, and near zero reads brutally: halving that chance would mean pinning the starting point down almost 24 orders of magnitude more tightly. Both numbers belong to the four-weight model. Neither is a measurement of a deep network.

What the deep networks show, and what they do not

For the real thing, the pair trained approximately 500,000 networks, spanning VGGNet, ResNet and BERT, image and text datasets, and a range of optimizers and training recipes, then ran the detailed diagnostics on one representative setup: a 12-layer VGG network trimmed to 12,036 parameters, learning handwritten digits with half the labels deliberately wrong. They picked it because it is cheap enough to sample finely, and reported that "the same qualitative results are observed across all configurations, including those without label noise."

Mapping where a grid of neighboring starting points ends up gives 1,772 distinct destinations, each a pattern of which neurons fell silent, interleaved rather than sorted into territories; magnify a patch and the interleaving is still there. The deep networks give a different exponent, and it is the one to carry: 0.000, give or take 0.002, flat across 10 orders of magnitude of starting-point precision. They measured it twice: on a graphics processor, whose arithmetic is fast and not quite repeatable, and on an ordinary CPU, where it is. The CPU run is the control that matters, because it "enables deterministic computation, ruling out the possibility that the observed irreproducibility is an artefact of accumulated random error." In the same network, flipping only the least significant bit of the starting parameters sent nearly half of the pairs, 0.477 of them, somewhere different.

The paper is exact about which half of this it is claiming. Its summary sentence reads: "Together, our results establish a robust scale-insensitive limit to reproducibility in deep learning and support riddled basin geometry as a candidate organizing principle underlying it." The limit is reported as measured. The geometry is offered as a candidate: for deep networks, the authors claim "robust signatures consistent with riddling" and riddling as "a potential dynamical mechanism," and the proof that riddling is what is actually happening exists in the four-weight model and nowhere else.

A test for a network nobody has analyzed

A limit in the weights would matter less if the models behaved the same anyway, and different weights can encode the same function. So they measured behavior too. Two networks trained from nearby starting points disagreed about which class an input belonged to on roughly 2.9% of inputs, and tightening the starting points did not shrink the disagreement: the functional exponent, the behavioral twin of the first one, also sits at zero within its error.

That is the usable part. Identifying a large network's symmetries is an open problem, so the authors propose a test that does not need them: a setup is in the riddling-like regime when the functional exponent is near zero and nearby starting points still produce substantially different functions. It is a measurement an engineer can make on an architecture nobody has analyzed, and the paper, its code and its data, are all public, so the check can be run rather than believed.

The work extends a line the two began in a 2025 paper on multifractal loss landscapes, and it carries an unwelcome corollary: in most of the configurations they tested, the learning rates that trained best were the ones where reproducibility fell apart. Turn the learning rate down and training becomes repeatable and worse. If the geometry holds, the choice is not between a careless pipeline and a careful one. It is between a model that performs and a model you can get back.