Skip to content
See the World Through ScienceA project of ALLATRA
Source: PreprintarXiv2 sources

A Second Lab Checked Whether a Language Model Can Hand-Write a Neural Network

By Wilkens EtienneWriterAI & Technology6 min read

Republish this story

Our work is licensed under Creative Commons BY-NC 4.0. You may republish this piece for free — with credit to ALLATRA Media and a link to the original, unedited beyond length trims, and not for commercial use.

Read the full license

The red brick British Library building in London, seen across its paved front plaza, with the Victorian spires of St Pancras station rising behind it.
The Alan Turing Institute, whose four researchers spent three days rerunning the experiment, is housed in this building in Camden (illustrative)."London: The British Library (Camden)" by michaelday_bath, via Flickr, CC BY-NC 2.0

Asked for a neural network that plays a miniature version of the arcade game Breakout, GPT-6 Astra typed out 20,486 numbers in a single reply, with no tools: no way to train the network, no way to run it, no chance to watch it play first. Loaded in, those numbers outplayed the reinforcement-learning baselines the researchers had trained for comparison. Rikiya Takehi and three colleagues at MIT's Computer Science and Artificial Intelligence Laboratory published that on Nov. 2, 2026, not as a paper but as a blog post.

The day after it appeared, four researchers at the Alan Turing Institute began checking. They worked for three days and posted the result on Nov. 16, 2026, as a preprint they label a working note of preliminary findings. Roksana Goworek, Alexander Hickey, Federico Nanni and Martin O'Reilly say they contributed to it equally. Their verdict is that the headline claims hold, and that both abilities are narrower than they sound.

Writing weights reproduced. Across three runs, Astra's Breakout networks scored 514, 550 and 307, against the 517 the original post reported, and on Asterix the Turing networks came in a little higher. On Seaquest they did not: 26, about half the 53.1 of the post's single run, and the one headline score that failed to come back. The team also closed the obvious loophole, commissioning networks for two Breakout variants that exist nowhere else, with left and right swapped or the board upside down. Those played as well as the standard game, while the standard-game network scored almost nothing on them. Weight reading reproduced too, with the game named every time and both models falling to chance on shuffled weights.

What those weights actually are

The experiment that explains the trick is the Turing team's own, and it is also the one that takes the shine off. The researchers gave two models the Breakout network Astra had written, with the game's rules, source code and input and output layout, and asked each to turn the weights back into Python. Neither could run the network or simulate the game. Both programs choose the same move as the network in all 49,189 game states the team tested, and over a thousand games they average 502.9, exactly what the network scores. Networks written by three weaker models could also be turned back into Python, and the Python copies repeated their mistakes exactly.

That is what the Turing authors mean by compiled programs, a phrase that is theirs and not MIT's, and its point is deflationary. Unlike a trained network, whose inner workings are hard to untangle, this one is just a pre-planned rule turned into numbers. That’s why two other models could read it back so easily. Across six models from two labs, each generation wrote a strong game-playing policy as Python code a generation before it could write the same policy as weights, so what is new is the translation step. The ability "currently amounts to compiling hand-designed programs and reading the layout of the input layer," the authors conclude, and that is their reason for rejecting the original's framing, in which the models natively understand weights.

Where the ability runs out

The compile-a-program account predicts where writing weights should fail, and it does. Asked for a network that classifies handwritten digits under the same conditions as the games, Astra's networks averaged 49.6% on the test images. A baseline that does nothing but compare each image with the average image of each digit reaches 82.0%, and the same architecture trained normally reaches 96.7%. Recognizing handwriting can’t be reduced to a simple rule, and that is exactly why the method failed here.

Reading has a matching limit. Every game network in the original takes the same grid of inputs, each channel labeled with the kind of object it holds, so the first layer is close to a picture of where a game keeps things, which a few lines of code turn into a heatmap a person can read, as the critic Sean Goedecke showed. The Turing team therefore trained policies for three games of its own, among them Whac-A-Mole. Astra recovered the layout in every run and mostly missed the mechanics: it read the whacking action as the one to avoid, never noticed that the ship in another game fires the hazards, and called a third game Frogger all three times. "But knowing where things are is not the same as knowing what a network does," the authors write.

The audit that half worked

Auditing a model without running it was the first application the original post proposed, and the post tested none of it. The Turing team did, in a setting where the model could read files but run no code. Astra picked out a small language model poisoned during training, to emit gibberish on a keyword, in all 10 runs, and one poisoned by later training in 7 of 10, which the authors say "does not provide good evidence that it is doing better than chance." In one test, Astra had to spot the harmful one of two model add-ons, and got it right 23 times out of 32. When the harmful add-on contained a hidden backdoor, though, it did no better than guessing, and neither did a smaller open model the team also tested. In none of 64 runs did it identify the trigger or describe what the sabotaged model does, which is the part an audit needs.

None of this is peer-reviewed. The Turing note is an early version posted on arXiv. The MIT work it checks is a blog post, and the authors themselves describe it that way in how they ask to be cited. What the story rests on instead is a second, unaffiliated laboratory running its own experiments and its own controls. That laboratory is candid about its three-day sprint: most conditions ran between one and five times, scores vary between runs, every network studied is small, and "the code was written primarily by Claude Opus 5.5 and GPT-6 Astra, the same model families we evaluate, and has not been independently reviewed."

The boundary matters, because the post's most spectacular results are not in the replication at all. The Turing team did not test the edit that lifted a small model's MATH-500 score from 7.9% to 48.1% by changing a fraction of a percent of its weights, nor the Go-playing network. It relays other people's objections to them: that the math gain comes from stopping the model's reasoning loops rather than teaching it math, and that "natively" reading weights covers setups that used Python, only three of the experiments working from raw weights alone on one critic's count. The MIT authors answered on Nov. 12, 2026, in a section added to the post, arguing that "each of our baselines tests only the narrow claim of its experiment." The MIT authors accept that the math gain came from stopping the loops. They also report trying the same thing on a model that doesn’t loop, and it made no real difference.

Two other people say they reproduced the original, both in posts on X, and the note cites them without vouching for either. The one outside result the team did check, a claim that a much smaller open-weight model reads game networks too, could not be reproduced at the reliability claimed: a little under 40% correct, above its shuffled control but not the clean result reported. What the three days leave is narrower than the week's excitement, and firmer. A general-purpose model, used off the shelf, can write a small working network from scratch, which the Turing authors call, as far as they know, a first. The networks it writes are programs in disguise. "For now, weight reading looks like a useful first look at a small network, not yet a way to tell what a trained model will do," they conclude.

Sources

Spot an error?

Spot an error?

Report an error

Spotted a mistake on this page? Tell us what's wrong and our editors will take a look.

What kind of problem?

Only if you'd like us to be able to follow up. We won't use it for anything else.

We correct mistakes openly. Select any text to flag it. Fixes are logged under our Corrections Policy.

Report an error

Reporting on

A Second Lab Checked Whether a Language Model Can Hand-Write a Neural Network

What kind of problem?

Only if you'd like us to be able to follow up. We won't use it for anything else.

We read every report. Corrections are logged publicly.