Skip to content
See the World Through ScienceA project of ALLATRA
Source: PreprintarXiv1 source

AI Agents Asked to Predict Lab Results Agreed With Each Other More Than With the Lab

AI & Technology

Republish this story

Our work is licensed under Creative Commons BY-NC 4.0. You may republish this piece for free — with credit to ALLATRA Media and a link to the original, unedited beyond length trims, and not for commercial use.

Read the full license

A scientist in a white coat places a tray of sample vials into a benchtop chromatography instrument on a dark laboratory bench.
A scientist loads sample vials into a benchtop chromatography system at Argonne National Laboratory in Illinois, photographed July 27, 2012. The new benchmark asks AI agents to predict outcomes that people at benches like this one had already measured (illustrative)."Materials Engineering Research Facility" by Argonne National Laboratory, via Flickr, CC BY-NC-SA 2.0, cropped

A research team has released a benchmark that hides the measured results of 4,207 real laboratory experiments and asks AI agents to predict them. In a preprint posted Nov. 11, 2026 and not yet peer reviewed, the team reports that thirteen configurations of frontier models agreed more closely with one another than with the measurements.

The benchmark, called the Science Prediction Universe, is aimed at the question a scientist faces before spending bench time. The paper argues that many questions in experimental science have no computed answer: software can narrow a list of candidate conditions, but which electrolyte additive or which enzyme variant wins is settled by synthesis and measurement. Each task hands an agent a procedure and a set of candidate conditions, and asks it to order them by a measured value it never sees.

On the paper's scale, a perfect order scores 1 and random guessing averages 0. The best of the thirteen reached 0.478 on the tasks drawn from published studies, though its provider's safety filters made it refuse most of the chemistry, biology and medicine tasks, and it was scored on fewer than half of them. Sixteen further tasks come from unpublished experiments run with partner laboratories; their results have been published nowhere, so no model could have been trained on them.

The team, which built the benchmark and ran the evaluation itself, reports that two configurations placed the same experiments in the same order at 0.52 on an average task, while their orders matched the measured ones at 0.33. It traces the gap to a shared habit: where candidate conditions differ along a number, such as temperature or concentration, the agents tended to rank along that number more often than the measurements do. On that reading, the limit sits in scientific judgment rather than in the agents' tool use.

The project team is based at UniPat AI; Zengji Tu is a core contributor and Liang Chen the project leader. The paper says the private tasks will stay unreleased, so the benchmark keeps working as the published tasks age into training data.

Sources

Spot an error?

Spot an error?

Report an error

Spotted a mistake on this page? Tell us what's wrong and our editors will take a look.

What kind of problem?

Only if you'd like us to be able to follow up. We won't use it for anything else.

We correct mistakes openly. Select any text to flag it. Fixes are logged under our Corrections Policy.

Report an error

Reporting on

AI Agents Asked to Predict Lab Results Agreed With Each Other More Than With the Lab

What kind of problem?

Only if you'd like us to be able to follow up. We won't use it for anything else.

We read every report. Corrections are logged publicly.