Close to a Third of a Leading AI Coding Benchmark Was Broken
A Terminal-Bench task called query-optimize asks an AI agent to make a slow database query run fast. Until the task was repaired, an agent could do exactly that and still score zero. The grading code compared a hash of the database file, so an agent that touched the data and then put it back produced a file that was logically identical and not bit-identical. Nothing in the instruction said the file had to survive byte for byte.
That task is one of 28 that E. Kelly Buchanan, Christopher Ré and colleagues at Stanford University and Cornell University repaired in a continuous audit of Terminal-Bench 2.0, a benchmark widely quoted in claims about how well AI agents write and fix code. Their audit paper was posted Nov. 5, 2026, as a preprint and has not been peer-reviewed. The repairs themselves shipped earlier in 2026 as Terminal-Bench 2.1, so what is new here is the measurement of what they changed.
Close to a third of the benchmark needed work, and the authors' argument is not that it was built carelessly. Terminal-Bench 2.0 went through human review, machine-assisted auditing and deliberate attempts to break its tasks before release. Within months, its issue trackers held dozens of reports of defective tasks anyway.
The authors call that accumulation benchmark maintenance debt, and they sort it into three kinds. Instructions that disagree with the tests were the most common: hidden requirements like query-optimize's, and tests so strict that they reject work the task description plainly allows. Task drift is the benchmark rotting from outside, as when one task's reference solution stopped working because a protein database changed the address it serves sequences from. Environment problems are the machine rather than the question: on one build task, the standard command for counting processor cores reported the host machine's count instead of the container's share, so an agent would start a parallel compile, run out of memory, and be killed before it could notice.
Only 14 of them were new to this audit. Eleven had already been flagged in an earlier review by Z.ai, the company behind the GLM models, and the rest came out of public issue reports and pull requests on the benchmark's own repository.
The ranking held, the scores did not
To find out what the repairs did to published results, the team re-ran 19 agents, each a model paired with the software harness that drives it, on both versions of the benchmark. The broad ordering barely moved: the rank correlation between the two leaderboards, the Spearman measure the authors use, came out at 0.958, close to a perfect match. Individual scores moved much more. They shifted by as much as 12 percentage points, and the agent that gained most, Anthropic's Claude Code harness running Claude Opus 4.6, climbed three places.
Twelve points is the maximum, not the typical case, and the direction was not uniform. On the per-agent table the maintainers published with the release, the average gain is about five points, and two of the agent and harness pairs came out slightly lower on the repaired benchmark than on the old one. The paper finds the same asymmetry: its two weakest agents both lost accuracy, which fits with the repaired tasks having been mostly hard ones that those agents rarely solved either way. The largest gains landed on tasks where the grader had been rejecting correct solutions.
So the rough picture of which agents are better survived the repair, while the absolute numbers quoted from the benchmark did not. For two agents a few points apart, which of them leads can depend on which version was run.
The clock is part of the measurement
Some failures had nothing to do with broken tasks. They came down to time limits. Every Terminal-Bench task comes with a time limit, and timeouts, rather than wrong answers, accounted for most failed attempts at the default setting. Swapping the harness around OpenAI's GPT 5.4 moved its score by about 20 points, a gap as wide as the one between some different models, and most of it closed when the team extended the time limit. The harness was not making the model more capable. It was making it faster. Resource budgets, the authors argue, are part of what a benchmark measures and have to be validated alongside its tasks.
The people measuring the repairs are the people who shipped them
This is a self-audit, but the acknowledgments don’t make that clear. The Terminal-Bench team's own release note for version 2.1 is signed "TB2.1 Lead: Kelly Buchanan", the paper's first author. The pull request that carries the 37 commits of repairs into the benchmark's repository was opened by her in March 2026, to show the diff and host discussion rather than to be merged. The paper's own acknowledgment thanks the Terminal-Bench team for reporting issues, which reads like the posture of an outside reviewer. It does disclose the circularity where it bites hardest: because the audit workflow helped build the repair set, the paper says, the scores it reports for the workflow itself "measure retrospective agreement with the release rather than independent defect discovery".
What keeps the result checkable is that little of it has to be taken on trust. The pull request lists every change, the paper’s tables cover all 28 repaired tasks with before-and-after pass rates, and the maintainers published the scores for each agent. The phenomenon has support from outside the group, too. Z.ai's review came first and from a different organization, with its own container fixes and its own diagnoses, and it reported that environment fixes alone lifted Claude Opus 4.5 by more than five points. OpenAI's audit of SWE-bench, an older coding benchmark, flagged more than a third of the original problems as underspecified. A benchmark needing this much repair is an ordinary finding, not a shocking one. The specific figures here, though, rest on this one paper.
The authors call Terminal-Bench 2.1 "a checkpoint rather than a terminal state", and it has already been passed by. In an August 2026 update, Z.ai said that models have overfitted to version 2.1 and are fitting incorrect solutions, and pointed users to a newer benchmark instead. The lesson is less tidy than a leaderboard. A benchmark score quoted months after release says something about how well the benchmark has been maintained, not just about the model.