Skip to content
See the World Through ScienceA project of ALLATRA

The Number Everyone Quoted, and the Evidence No One Can Check

By Olga SchmidtEditor-in-Chief, WriterAI & Technology7 min read

Republish this story

Our work is licensed under Creative Commons BY-NC 4.0. You may republish this piece for free — with credit to ALLATRA Media and a link to the original, unedited beyond length trims, and not for commercial use.

Read the full license

A narrow walkway between two tall rows of server racks in a computing facility, cabinets stacked from floor to ceiling on both sides.
Server racks at a research computing facility near Geneva, photographed Sept. 15, 2019. No public picture shows the systems or the internal records this assessment examined (illustrative)."CERN Computer Center 07" by SimonWaldherr, via Wikimedia Commons, CC BY-SA 4.0, cropped

Nothing special happened on Oct. 7, 2025. It is simply the date a statistical model picked when it was given Anthropic’s benchmark scores and asked to fit two straight lines instead of one, with the bend wherever the data suggested. After that date, the line is about 1.9 times steeper than before. That ratio is the number that made headlines.

The fit comes from the nonprofit METR, whose staff spent eleven weeks inside Anthropic, from Aug. 12 to Oct. 29, 2026. Four of them (Tom Cunningham, Nate Rush, Parker Whitfill and Hjalmar Wijk) were given internal benchmark scores for models the public has never seen, training-compute metrics, staff surveys and about fifteen interviews. What they published on Nov. 5, 2026, is not a peer-reviewed study and not a paper, but a self-published assessment report whose roughly 100-page full version is withheld.

What rose 1.9 times, with a 95% confidence interval of 1.5x to 2.8x, is the growth rate of AECI. That is Anthropic's in-house version of the Epoch Capabilities Index: one score fitted across dozens of benchmark results, computed here with Epoch's public code. The models in the fit are Anthropic's alone, four of them never released. "We use 'acceleration' to refer to an increase in the rate of growth, such that a 2x acceleration means that 2 years of historical progress is made in 1 year," METR writes. It measures neither research output nor capability in the world: "We believe there is significant risk that acceleration in AECI may not map cleanly to acceleration in real-world impact."

METR's own conclusion is weaker than its arithmetic. "As of October 2026, Anthropic is likely improving the capabilities of their models on well-specified tasks at around twice the rate they did between March 2024 and October 2025," it says, and then: "This estimate is given with high uncertainty; we would not be surprised if the true acceleration was closer to 1.3x or 3x the historical rate." The report uses three different time periods, and they shouldn’t be confused. METR’s baseline runs from March 2024 to October 2025. The AECI fit breaks at the learned Oct. 7, 2025. A third comparison has Anthropic making "as much progress on benchmarks in the past year as they did between October 2023 and October 2025."

How one assumption changes the result

The report prints a range: "Robustness analyses find acceleration rates between 1.1x and 2.9x." Its main text credits that to choices about which benchmarks and models to include; the appendix shows otherwise. Changing the snapshots, the benchmarks or the regularizer moves the estimate only between 1.9x and 2.2x. The extremes come from a choice the main text does not name: the link function, which is an assumption about how hard the tasks inside a benchmark are. Assume a long tail of easy problems and the answer is 1.1x, with an interval of 0.8 to 1.5 that includes no acceleration at all. Assume a long tail of hard ones and it is 2.9x. All of them predict held-out data within about a point of each other, and METR does not choose: "The quality of fit is fairly similar across each of these link functions (except cauchit), and we do not have strong a priori reasons to prefer one."

Then there is the denominator. Fitted over all frontier models instead of Anthropic's, the same analysis gives less, and the report prints two readings without picking one. First: "Anthropic models were subfrontier in 2025, so Anthropic's historical progress up to 2025 was understated. Therefore, the 1.9x acceleration numbers in the rest of the report are overstated, and 1.4x is more accurate." Second, that the historical trend is the overstated part, which leaves 1.9x closer to right. Anthropic's Claude Haiku 5.5 system card reported 1.7x, on METR's account of it, and METR puts the difference down mainly to its own inclusion of newer unreleased models.

How much of it is the AI

The second question METR's eleven-week assessment inside Anthropic set itself was attribution, and there the report says outright that it is guessing. "We guess that a majority (perhaps ≈80%) of the current (October 2026) acceleration is due to AI," it reads, adding that "This judgment is given with substantial uncertainty." Three things narrow it. The figure is about the current, instantaneous rate, not the cumulative share of everything since 2025, which METR expects to be lower. "The acceleration in late 2025 was likely influenced by non-AI factors." And underneath it, "there was not a consensus among Anthropic's own staff and we did not find decisive evidence," while the detailed estimates were redacted. Of eleven researchers whose interview quotes were coded, two were read as supporting a doubling from AI, three as leaning that way, four as unclear and two as against.

One finding is firm, and it is a negative: "We are confident that the acceleration was not due to an acceleration in the scaling of training compute." The evidence for it was redacted. The other figure that traveled runs the opposite way. Anthropic staff vote with Slack reactions on how much faster the newest model makes them, and average self-reported uplift reached about 12x in October 2026. METR calls that "only very loose evidence" from "an unrepresentative subset of staff," then makes the arithmetic point itself: "Note that individual uplift/speedup estimates do not correspond to acceleration rates." Its analysis of how much growth in uplift would be needed to explain the measured acceleration was redacted as well.

The evidence that stayed inside

The report's own opening says what the opacity cost. "While we were able to freely share our highest-level conclusions about risk, Anthropic redacted a majority of the evidence we found most informative for these conclusions." And then: "As a consequence, the public has to rely on METR's judgment much more than would be ideal, given the weight of the questions answered here." Of seven kinds of evidence listed as shared with METR, four could not be described publicly at all, among them intermediate capability measures and algorithmic progress estimates. Four further things it could not communicate are named, including its per-stage estimates of acceleration and its forecasts for future models.

METR’s own checklist is even more direct. Against the requirement that "The evaluator shared sufficient methodological details to allow for independent review of the results," METR's verdict is No, with the note "This was not possible due to redaction requirements." Query access, direct access to Anthropic's systems, up-front publication rights, redaction authority and a responsible disclosure policy are all No as well.Other parts of the arrangement point the other way. "No fees are paid in either direction"; the contract leaves METR sole discretion over methodology and conclusions; "Anthropic did not have editorial control over the final report, but did have control over what non-public information METR could incorporate into it." Conflicts are disclosed at length, down to the observation that "METR's work with nonpublic models and use of free tokens incentivize a cordial relationship with AI companies." Anthropic supplied the laptops and the tokens, approved every piece of information published, and "changed the scope of this project midway to exclude compute verification," the component that would have checked how much compute goes to capabilities and how much to safety.

None of this surprised the evaluator. Three weeks earlier, on Oct. 14, 2026, METR had written that "Even with transparency about limitations, it is possible for a company to use its rights to withhold access or redact to exert control over what evidence the public receives from us." The page has been corrected once since publication: on Nov. 6, 2026, the checklist's access item went from Yes to No.

A number somebody promised to act on

The reason the figure matters at all appears in METR's companion post of the same date. Anthropic's Responsible Scaling Policy, version 3.4, carries an early-warning threshold at a doubling of the pace of progress, triggered "If there is 'dramatic acceleration' of the pace of AI progress for reasons that likely relate to the automation of AI R&D." Reviewing the company's August 2026 risk report, METR writes: "While both METR's researchers on this project and Anthropic's relevant staff lean toward it not having been crossed, we think it is plausible that it had been, and we believe the public report overall understates the level of acceleration that was occurring." The threshold is the company's own policy and not a law. The company is also the only party in a position to show anyone the evidence either way.

Sources

Spot an error?

Spot an error?

Report an error

Spotted a mistake on this page? Tell us what's wrong and our editors will take a look.

What kind of problem?

Only if you'd like us to be able to follow up. We won't use it for anything else.

We correct mistakes openly. Select any text to flag it. Fixes are logged under our Corrections Policy.

Report an error

Reporting on

The Number Everyone Quoted, and the Evidence No One Can Check

What kind of problem?

Only if you'd like us to be able to follow up. We won't use it for anything else.

We read every report. Corrections are logged publicly.