The Top of the AI Coding Leaderboard Is Too Close to Call

Two entries sit at the top of SWE-bench Verified, the scoreboard quoted most often when one AI coding agent is said to beat another. Each resolves 396 of the benchmark's 500 real GitHub issues. The entries behind them are a handful of issues back. Five researchers asked the question the table invites and never answers: does a gap that small mean anything?

Their answer, posted to arXiv on Sept. 15, 2026, is that it does not. Fengshuo Liu of Imperial College London and four colleagues at Hong Kong Polytechnic University, Korea University, and Jinan University took the leaderboard's own verdict on every problem for every entry and ran paired statistics over them. They trained nothing and ran no model. On the Verified list, an exact paired test, which compares two entries only on the problems where they disagree, separates none of the 29 neighboring pairs in the top 30.

Which leaderboard this is matters, because the finding is easy to map onto the wrong systems. The pool is the set of entries teams submit voluntarily to the official SWE-bench site, each published together with the grading harness's verdict on every individual problem. Since November 2025 the maintainers have accepted Verified submissions only from academic teams and research institutions that open their methods. The SWE-bench Verified percentages that AI companies quote in their own announcements are not in this pool, and they run well above the 79.2% at the top of the submitted table.

The leaders are solving the same problems

Take the 10 leading entries together and 285 of the 500 problems are resolved by every one of them, while 51 are resolved by none. Only 164 are left that can tell any of the 10 apart. The rest still count toward every displayed score, which is how a benchmark can look healthy in the aggregate and be nearly silent at the point where people actually read it.

Nor is the overlap the useful kind, where different systems are good at different things. When two leading entries differ, the weaker one's successes are almost entirely contained in the stronger one's, well past what their scores alone would predict. The authors call that nesting, and it is what removes the resolution: pooling the top two adds a little, ordering them does not.

That makes the result narrower than a verdict that the leaderboard is useless, and the authors keep it there. Among the 435 possible pairings inside the top 30, 194 do separate with no correction applied, and 41 survive a stricter adjustment for testing that many pairs at once. A leader is still distinguishable from a mid-table entry. What the evidence cannot do is tell first place from second: across the 29 adjacent pairs the count is zero either way, and the closest of them sits nowhere near the threshold.

The number is not the model's alone

A leaderboard entry is not a model. It is a model together with its scaffold, the surrounding program that reads the issue, calls tools, retrieves code, and decides when to stop. The table names neither factor as such. Reconstructing the pairs from public metadata, the audit found one model, claude-3-5-sonnet, submitted with nine different scaffolds and scoring anywhere from 168 problems to 317. That spread of 29.8 percentage points is more than three times the 8.8 points covering the entire top 30.

Teams choose a model and build a scaffold around it together, so effort, access, and engineering are tangled into the same number, and more than half the Verified submissions carry metadata too loose to place in the comparison at all. Their fitted ranges for the two factors are "of the same order," they write, a correction aimed at anyone about to conclude that the scaffold matters more than the model. What the data supports is the weaker and more useful claim: the number belongs to the pair, not to the model.

Where the same test still works

On the far larger Test split, whose leading entries still span a wide range of ability, the same statistic separates 14 of 23 neighboring pairs. Irresolvability, the authors conclude, is a property of a converged comparison set rather than of the benchmark family. They also credit the thing that made the audit possible at all: SWE-bench publishes the per-problem verdict for every submission, a practice they call "not universal among leaderboards."

The paper makes the distinction explicit: "non-rejection does not establish equivalence." A test that fails to separate two agents has not found them equal. The paper repeats the point three times, sets no margin at which it would call two systems the same, and labels its own groupings descriptive rather than a ranking.

All of this describes the leaderboard as it stood on July 30, 2026. It has moved since. The maintainers folded the bash-only runs into Verified on Sept. 1, 2026, and a new submission arrived with them, so the specific table in the paper is a snapshot, while the structure it describes is not tied to that table.

The recommendations follow from the arithmetic rather than from indignation. Report how many problems actually discriminate a given comparison. Record the model and the scaffold in a machine-readable field. Publish tiers together with the rule that produced them, instead of strict ranks. And judge a new candidate problem by whether it breaks the nesting: more problems of the kind already there would take roughly 26,000 of them to separate a typical adjacent pair, while far fewer that disagree and lean one way would do it. For a company picking a coding model for its own work, the advice is to measure the pair it will actually deploy, because the ordering of scaffolds seen here reverses from one model to the next.