AI Agents Asked to Predict Lab Results Agreed With Each Other More Than With the Lab

A research team has released a benchmark that hides the measured results of 4,207 real laboratory experiments and asks AI agents to predict them. In a preprint posted Nov. 11, 2026 and not yet peer reviewed, the team reports that thirteen configurations of frontier models agreed more closely with one another than with the measurements.
The benchmark, called the Science Prediction Universe, is aimed at the question a scientist faces before spending bench time. The paper argues that many questions in experimental science have no computed answer: software can narrow a list of candidate conditions, but which electrolyte additive or which enzyme variant wins is settled by synthesis and measurement. Each task hands an agent a procedure and a set of candidate conditions, and asks it to order them by a measured value it never sees.
On the paper's scale, a perfect order scores 1 and random guessing averages 0. The best of the thirteen reached 0.478 on the tasks drawn from published studies, though its provider's safety filters made it refuse most of the chemistry, biology and medicine tasks, and it was scored on fewer than half of them. Sixteen further tasks come from unpublished experiments run with partner laboratories; their results have been published nowhere, so no model could have been trained on them.
The team, which built the benchmark and ran the evaluation itself, reports that two configurations placed the same experiments in the same order at 0.52 on an average task, while their orders matched the measured ones at 0.33. It traces the gap to a shared habit: where candidate conditions differ along a number, such as temperature or concentration, the agents tended to rank along that number more often than the measurements do. On that reading, the limit sits in scientific judgment rather than in the agents' tool use.
The project team is based at UniPat AI; Zengji Tu is a core contributor and Liang Chen the project leader. The paper says the private tasks will stay unreleased, so the benchmark keeps working as the published tasks age into training data.
Sources
- arXivPreprint
