Skipping Layers at Random Can Make an AI Model Cheaper to Train, and Faster to Run

Somewhere in the last few years, the people who train the largest language models quietly deleted a line from the recipe. Dropout is a breaking part of a neural network on purpose during training, so the rest learns to cope without it and it has been in the toolkit from the start. As models and datasets grew, it faded out, on the reasoning that a network swimming in fresh text hardly needs a trick invented to stop small models from memorizing small datasets. Nine researchers, led by Mostafa Elhoushi with the Cerebras Systems scientist Joel Hestness as senior author, now argue that the deletion was a mistake, and they presented the case at the International Conference on Machine Learning in July under a title that leaves little doubt: "Don't Drop Dropout."

The version they are defending is the blunt one. Ordinary dropout switches off scattered individual connections; layer dropout throws away whole layers, so that for one training step the signal jumps clean over a block of the network and the layers on either side have to make sense of the gap. It was ordinary practice in the transformer networks behind today's language and vision models, before training at billion-parameter scale took over: it made training faster, models a little more accurate, and finished networks tough enough to have layers pruned out afterward without falling apart.

What the ICML paper adds is a measurement of that decision at scale. Get three things right together — which layers are eligible to be skipped, how the skipping is scheduled across a training run, and how the optimizer is tuned around it — and a model can reach lower or similar validation loss, its score on text it never trained on, while saving up to 20% of the arithmetic the run costs. Up to, and lower or similar: this is the best case under a configuration the team went looking for, not a discount that arrives when you switch the feature on.

The second half of the claim is about what happens after training. A network that spent its training with layers randomly missing turns out to tolerate their absence when it runs, which opens three shortcuts: stopping early at a middle layer when the answer has already settled, skipping middle layers outright, and self-speculative decoding, where the model drafts an answer with part of itself and checks it with the whole. Together those are worth up to a 1.4x speedup with, in the paper's words, negligible accuracy loss. For anyone whose bill is dominated by running a model rather than building one, that is the more interesting number, because it is a property the model keeps for the rest of its life.

The authors write that no comprehensive study has quantified, let alone mitigated, the effect: a claim about pretraining large language models, not about dropping layers in general. That ground has been worked before: Minjia Zhang and Yuxiong He reported at NeurIPS 2020 that a progressive layer-dropping schedule cut roughly a quarter off both the arithmetic and the wall-clock time of training BERT, an earlier language model, with the resulting models doing as well on later tasks. What was missing is the same accounting at the scale where the technique was dropped.

All the pretraining experiments were run on Cerebras CS-3 systems (the hardware built by the company several of the authors work for, Hestness, Nolan Dey and Shane Bergsma among them). That does not make this a vendor benchmark. A saving counted in arithmetic operations does not depend on whose chip performs them, and the work cleared peer review at ICML, one of the field's main conferences. It does mean that the recipe has been demonstrated once, by one group, on one company's machines, and that nobody outside that group has yet run it and reported the same curves.

That combination gives real results: a single source, no outside replication, that is unusually easy to act on, because layer dropout changes the training loop and not the architecture. The finished model has the same shape either way, so a lab can run the schedule on something small, compare its loss curve against the paper's, and find out cheaply whether the effect survives on hardware nobody at Cerebras built. Elhoushi and his colleagues are explicit that the saving depends on getting the layer distribution, the timing and the optimizer settings right together, and they write the paper as a set of best practices for doing exactly that.