When Training Text Is Reused, the Newer AI Design Suffers First
Every large language model is trained on a finite pile of text, and the pile is running low. When the fresh human writing runs out, the cheapest thing left to do is show the model text it has already read, and the field has come to treat a few extra passes as close to free. A group at Stanford University and the University of Washington ran a grid of training experiments to check whether that still holds for the kind of model almost everyone now builds. Their answer, posted Sept. 10, 2026: it does not.
The free-passes number most people carry comes from dense models, the older design in which every parameter fires on every word. Working with those, Niklas Muennighoff, Colin Raffel and colleagues reported in the Journal of Machine Learning Research that up to four epochs of repeated data cost essentially nothing against training on all-unique text.
The models being built today are mostly not dense. A Mixture-of-Experts model splits each layer into many small sub-networks, called experts, plus a router that sends each word to only a few of them. The model can then carry an enormous number of parameters while paying to run a fraction of them, which is why the design took over. It is also, the new preprint reports, the design that suffers first when text is reused.
Atindra Jha and Margaret Li, who led the work with Jure Leskovec, Percy Liang and Luke Zettlemoyer, trained dense and sparse models from 80 million to 1 billion active parameters. Every model got the same total number of training tokens; what changed was how many of those tokens were unique. At the smallest scale, dense models took eight passes over the same text with only a slight rise in error on text they had not seen, while the sparse models took a visible hit at four. Sparse still won outright up to 16 passes. At 32 the order reversed, and the plain dense models came out ahead.
The tempting reading is that routing is fragile. The paper's own numbers say something narrower and more useful. Because the comparison holds active parameters fixed, every sparse model in it carries far more total parameters than the dense model it is measured against, and the damage tracks total parameters rather than active ones. A coincidence of sizes shows this cleanly: the repetition curve of a 200-million-parameter dense model falls between those of two sparse models with 80 million active parameters, one holding 158 million parameters in total and the other 244 million. Sparsity is how the extra parameters get there cheaply. It is not a defect in the routing.
The fix works, and it has a price
Having found the failure, the team worked through the standard remedies for overfitting, or memorizing the training data. Weight decay did nothing measurable. Gradient clipping did nothing measurable. Router jitter, the one remedy aimed at sparse models specifically, did nothing measurable either. What worked was the family of methods that switch parts of the network off at random during training: ordinary dropout, and several variants that blank the output of the experts. Turn the dropout rate high enough, and the sparse models beat dense ones even at 64 passes over the same text.
Two conditions travel with that result. The same high dropout rate hurts performance when repetition is low, so it is a setting to reach for only once the text really has run short. And no method the team tried matched training on all-unique data. These remedies buy back part of the loss, never the whole of it.
Not the router, the experts
The team also looked inside the models for the thing that breaks. The first suspect was the router. Tracking which expert each word went to across saved snapshots of training, they found that routing settles almost immediately: assignments start near random, and by a tenth of the way through training most words already keep the same expert from one snapshot to the next. By the end they barely move. Repeated data makes the router settle harder still.
Then the story turns. Models trained with dropout showed less of that hardening at 64 passes, and they overfit far less, which is backwards if a stuck router is the cause. From that, the authors hypothesize, the overfitting is "not primarily driven by routing, but rather the functions learned by each expert." Their measure of that is expert knockout: switch a single expert off when the finished model runs and see how much worse it gets. Going from one pass to 32 more than doubled that cost in models carrying 128 experts, and dropout brings it down. Each expert, on this reading, is memorizing its own small shard of a small corpus. The link is a correlation rather than a demonstrated cause, but the remedies that work share one property: they stop any single expert from being indispensable.
The largest model here is a thousandth of a frontier system
Whether any of this survives at the scale the industry actually ships is the question the paper cannot answer. Its largest model has 1 billion active parameters and 8.5 billion in total, two to three orders of magnitude below frontier systems, and the work is a preprint that has not been peer reviewed.
The direction, though, is one a peer-reviewed result already pointed in. Fuzhao Xue, Yang You and colleagues reported at NeurIPS in 2023 that the damage from repeated passes tracks parameter count rather than the arithmetic cost of a forward pass, and that of the usual remedies only dropout helped, its rate needing careful retuning as models grow. The new paper notes that the earlier one reached its parameter conclusion from a single sparse configuration; this one sweeps three model scales and a wide range of expert counts and sizes.
For anyone planning a training run, that turns a vague worry into an arithmetic one. What matters is how many parameters are being fit to each unique token, and a sparse model quietly puts a much larger number on top of that fraction.