AI Agents Got Better at Clinical Data Tasks. Most of It Came From Writing the Rules Down.

Between a hospital's raw records and a published medical study sits a step almost nobody writes about. The database holds lab values, drug charts, timestamps and item codes. The study needs a variable: whether this patient had a suspected infection, how much urine they passed in each rolling window. None of that is a column. Somebody has to decide which item codes count, which window applies and what to do about a missing measurement, and those choices change who ends up in the cohort.

A benchmark posted on Oct. 27, 2026, by Hannes Ill, Leo Anthony Celi and colleagues at MIT's Laboratory for Computational Physiology asks whether an AI coding agent can rebuild those choices. M4Bench sets 28 derivation tasks on MIMIC-IV and eICU, two de-identified critical-care databases that researchers can reach only with credentials. The agent gets the task and a copy of the database with the answer table deleted, and the file it writes back is scored against the definition a human research group already published. Each run sits in its own container, and the harness fails closed if the agent can reach the hidden answer. The paper has been accepted for presentation at NeurIPS 2026, in its Evaluations and Datasets Track, and the authors posted this camera-ready version themselves; the proceedings do not appear until December 2026.

The scoring quantity matters more than the number. The authors call it reward: the share of the reference answer's value columns the agent got right, averaged over columns on a scale from zero to one, with rows the agent missed, invented or duplicated counted against it. It measures partial agreement with somebody else's published definition, nothing more. On that measure, in a Codex campaign using GPT 5.4 Mini and GPT 5.5, agents handed a clinician-written procedure note for their task came out 0.136 above agents handed none, and improved on 26 of the 28 tasks.

The controls were built to ask why

Most benchmark papers would stop at that number. This one had built the apparatus to ask where it came from. On an eight-task subset, the team swapped the curated note for what they call an operational specification: the complete clinical rules written out in prose, with the item identifiers and the dataset recipes stripped out. The knowledge, in other words, without the navigation. Reward on those eight tasks rose by 0.162. The curated note, added on top of that, bought a further 0.024, an amount whose uncertainty range runs from slightly negative to about a tenth of a point. The paper's word for the result is most, and it uses that word in all four places it states the finding: "Specification completeness accounts for most of this gain," the abstract says.

Those eight tasks were fixed in the benchmark matrix before any of them ran, which rules out picking them to suit the answer. They are still not a neutral eight. They over-represent the tasks where the notes helped most, and not one of them comes from eICU. The authors put the limit in five words of their own: "Mechanism conclusions apply to these subsets."

Two further controls close off the easy alternatives. A decoy note, a procedure written for a different task altogether, scored the same as no note at all, so extra text in the window was not what helped. And the reference implementation's own SQL, handed over as text the agent still has to adapt, because some of the tables it mentions are not in the task database, scored almost exactly what the curated note did.

Almost the same score, nowhere near the same answer

That last pair is the sharpest thing in the paper. On the ten tasks where both controls ran, raw reference SQL and the curated notes land almost on top of each other on reward, at 0.837 and 0.833. Then the evaluator's strict checks are applied, and raw SQL passes every one of them on 49 of 100 runs against the notes' 7 of 100. The same headline agreement; a sevenfold difference in actually reproducing the reference.

A strict pass means nothing was wrong: every required column present, the expected set of rows covered, every scored column inside its tolerance. In the primary comparison it almost never happened. Only 25 of 280 runs with the notes cleared that bar, against 13 of 280 without, and 280 counts runs rather than tasks, because each task was attempted five times by each model. Nor did the notes change which rows came back: the row-level agreement measure moved by 0.003. The failures mostly shared one shape, with the agent finding the right patients and putting a wrong number beside them.

The paper spends a section on what it does not show

Several limits the authors state bear on how far a reader should carry the result. The scores measure agreement with reference implementations taken from public concept code, not clinical correctness. The authors' own clinician review flagged places where those references are arguable: broad culture inclusion can overcall suspected infection, the organ-failure score treats a missing component as a zero, the vasopressor mapping leaves out angiotensin II. The notes were clinician-reviewed, but the reviewing clinicians are co-authors, and the paper says of its own review that it "is not external validation."

All but two of the 28 tasks come from MIMIC-IV, and the two eICU tasks are cross-database probes, not evidence that this travels between hospitals. Several of the reference implementations use information that only exists after the fact, such as discharge codes, so this is a test of reconstructing variables in hindsight, not of prediction at the bedside. The same team put the whole task set through a second family of agents, Claude Code, and got the same direction at a slightly smaller size. That is a consistency check rather than an outside one: same authors, same tasks, same evaluator, and the paper calls it descriptive.

What M4Bench leaves behind is less a verdict on agent skills than a format for arguing about them. The tasks, the notes, the evaluation code and sanitized per-run logs are released openly under a permissive license; the task databases are not, because MIMIC-IV and eICU come through PhysioNet under credentialed access, so another group has to obtain its own and rebuild them from the released setup scripts. The authors' advice to anyone pointing an agent at this kind of work is procedural rather than reassuring. Declare the output grain, the keys, the window boundaries, the units and the missingness rules before any code is written; check all of them afterward; and treat what the agent returns as "an untrusted draft that needs independent clinical review before consequential use."