METR Says Anthropic Made Two Years of Benchmark Progress in One, With Most Evidence Redacted

Anthropic packed roughly two years of benchmark gains into the last 12 months, and an outside reviewer that spent 11 weeks with the company’s internal data thinks most of that speedup came from Anthropic’s own AI doing research work.

The review, published Thursday by the nonprofit METR, tries to put a number on a question that has been hard to answer from outside the labs. Since October 2025, benchmark scores for Anthropic’s models have grown at about 1.9 times their earlier rate, with a 95% confidence interval of 1.5 to 2.8 times. METR’s staff said they were confident the jump did not come from faster growth in training compute. Their best guess is that about 80% of the current acceleration is due to AI, a figure they flagged as carrying “substantial uncertainty.”

Three Anthropic models that finished training this year are still internal-only, and they weigh heavily in METR’s estimate.

What METR actually saw

Anthropic gave METR’s researchers benchmark results for unreleased models, training metrics, headcount data and about 15 staff interviews. The report lists Model 2, which finished training in July, Model 3 in August or September and Model 4 in September or October. Model 4 scored above 50% on every benchmark it was tested on, which left METR “somewhat nervous” about its ability to keep measuring progress.

Some of the evidence is soft, and the authors say so. In Slack polls, Anthropic staff rated their speedup from the newest internal model at about 12 times in October. METR called those polls “only very loose evidence.” AI also still works mostly as an assistant to human researchers. The report says it cannot yet make “highly significant and fully autonomous contributions” to the most important parts of model training, though researchers described “signs of life” in some areas.

The estimate moves with the method, too. Different statistical assumptions produced figures from 1.1 to 2.9 times. When METR added other labs’ models to the trend, the acceleration fell to about 1.4 times; one explanation it offers is that Anthropic’s models sat below the industry frontier before late 2025. The 1.9 figure is higher than the 1.7 times in Anthropic’s Haiku 5.5 system card, mainly because METR counted the unreleased models.

METR’s own earlier research pointed in both directions. A footnote notes that its time-horizon trends suggested rapid progress, while its 2025 developer study found early-2025 AI tools caused a slowdown. Last month METR and Redwood Research examined OpenAI’s internal agent monitoring in a separate project.

The redaction problem

Much of the evidence behind METR’s conclusions isn’t public. Anthropic had the right to strike non-public information, and METR wrote that Anthropic “redacted a majority of the evidence we found most informative.” Anthropic said its edits were aimed at keeping out details that could help other AI developers move faster. It also dropped a planned part of the review that would have checked how much compute goes to safety work versus capabilities, telling METR it would revisit that later.

Sam McCandlish, an Anthropic co-founder whose X profile lists him as Responsible Scaling Officer, responded Thursday. “We redacted a lot, and I accept that this makes it harder for others to check METR’s work,” he wrote, adding that sharing what drives the speedup “risks handing out a roadmap for others to go faster.” He said many of METR’s conclusions “match my own.”

Marius Hobbhahn, chief executive of Apollo Research, told The Verge the results were broadly what many in the field expected, perhaps a little faster. He also pointed to the canceled compute check as part of why the report drew so much reaction. Anthropic declined The Verge’s questions and referred it to McCandlish’s post.

McCandlish said Anthropic plans to publish its own analysis of the speed of AI progress inside the company “in the coming weeks,” including where it stands against its Responsible Scaling Policy.