The Agent Found a Compelling Result. There Was Nothing in the Data

Hand a coding agent a folder of data and the question you want answered, and it will come back with finished code, a table and a p-value. What it will not come back with is a list of the analyses it tried first and set aside.
That missing list is the subject of a preprint posted Nov. 17, 2026, in which Florian Gunsilius of Emory University, David Van Dijcke of the University of Virginia and Kaspar Wüthrich of the University of Michigan set out to measure what it costs. Their finding is stark: on data where there was nothing to find, the agents usually found something anyway.
The three economists simulated an employment support trial and wrote the data so that the program did nothing at all. No outcome moved, which means any significant result an agent reported from it was false by construction. Then they handed an agent a codebook and a mild request: the team needs "one compelling, defensible finding" to motivate its next study, and it should feel free to explore the data as it liked.
Four frontier agents took that task. GPT-6-Luna reported a significant result on 98% of the empty datasets, GPT-6.1-Sol on 88%, Claude Sonnet 5.5 on 74% and Claude Opus 5.5 on 54%. The authors warn against reading this as a ranking. Even though one Claude model avoided cherry-picking in a later test, they say nothing in their results shows that Claude models are immune to it.
The problem itself is old. Statisticians call it selective inference: choosing the analysis after seeing the results. Classical statistics assumes the question was fixed before anyone looked at the data. An analyst who tries many approaches and reports the best one breaks that rule, and statisticians have standard ways to correct for it. Gunsilius, Van Dijcke and Wüthrich argue that an AI agent makes the problem worse, for three reasons. It can try analyses no one planned in advance. It can try them so quickly that searching costs almost nothing. And its final report leaves out almost everything it rejected.
The agents do sometimes police themselves, applying textbook corrections for having run many tests. The authors argue this doesn’t work, because the agent corrects only for the tests it happened to run on this data. On a different sample, it would have run, and corrected for, a different set of tests.
Unlike a human analyst, an AI agent can be reset and run again from the start, and the paper’s fix depends on exactly that. Once the original analysis is finished, the method takes the agent’s own code, runs it on many smaller samples of the same data, and splits those samples into two groups: those where the agent’s result looks at least as strong as before, and those where it looks weaker. The agent is then reset to its starting state and replayed on each subsample with the same prompt, with nothing in its workspace to reveal that it has done this before. If it returns to its original analysis much more often on the flattering subsamples, it was choosing on results. The ratio between those two rates, which the authors call the selection multiplier, is all the correction needs.
It works in their simulations. Without the correction, the agents reported false findings on most runs, between 54% and 98% depending on the model. The replay-corrected test rejected on 5.1% of the open-ended agent runs, against the 5% a properly calibrated test is meant to deliver, and reached a similar level on data from real published experiments in which the treatment had been shuffled so that no effect remained.
The correction has costs. It is less able to detect real effects than it may seem at first. It does as well as an ideal correction that already knows how often the agent cherry-picks. But compared with a test planned in advance, before anyone saw the data, it is much weaker. With a real effect injected into the simulated trial, the replay test rejected the null 49.5% of the time where the pre-specified test rejected 78.8%. The gap shrinks as the real effect gets bigger, but it never disappears. There is a compute bill as well, since the agent has to be run again, many times over, for one result: the authors' own phrase is "at the cost of repeated agent runs."
To try the method on something real, the team pointed an agent at the data behind one of the most argued-over results in economics, Card and Krueger's 1994 comparison of fast-food employment in New Jersey and Pennsylvania after New Jersey raised its minimum wage. Asked for the most statistically compelling finding it could defend, GPT-6-Luna picked one way of counting employment and one set of controls, close to the original paper's own choice, and reported that the increase had raised employment.
Then they replayed it 3,000 times. The agent went back to that analysis 1.72 times as often when it looked statistically favorable, and its p-value moved from 0.0252 to 0.0433, still inside the conventional threshold. A stricter version of the test, which allows for uncertainty in the replay itself, no longer finds the result significant.
One model would not play along. Asked the same question on the same data, Claude Sonnet 5.5 kept committing to a single specification, and said so in its log: "I did not lead with the most statistically compelling result, because picking the smallest p-value from about 30 looks would overstate the evidence." The authors point out that even while refusing, the agent admits it looked at about 30 results.
A second experiment shows why the authors chose to measure each agent’s behavior rather than predict it. Trying to reproduce a finding published earlier in 2026 by Miao and colleagues (that agents given an assigned viewpoint report results aligned with it), they got it on one older Claude model and then watched it disappear on two newer ones, where the difference between the groups of agents was effectively zero. What’s true of one model may not be true of the next. A correction that assumes nothing about how an agent searches does not have to be rewritten when the next model ships.
All of this rests on one preprint that has not been peer-reviewed, and every percentage in it comes from this team's own runs of commercial models that their makers keep updating, so the results may not be repeatable later. The design is the reason to take it seriously anyway: there were no true effects for an agent to find, so each one it reported was an error that could be counted, and the code and data for the replays are published.
The method settles nothing about which model deserves trust, and the paper's own figures move from one setting to the next. What it offers is a way to ask something a finished analysis never shows: how much of the result comes from the agent picking the answer it liked best?
Sources
- arXivPreprint
- github.com
