Show an AI Judge a Previous Score, and It Marks the Same Text Lower

Ask a language model to grade a piece of writing twice, and the second grade is supposed to be a fresh one. Five researchers at Infobip, a messaging company with offices across Croatia and Bosnia, tested that assumption by leaving one extra line in the prompt (the score a previous judge had supposedly given the same text) and found the new grade sliding toward it. It happened in seven of the eight models they tried, and in GPT-4.1 and Claude-4.5-Sonnet it pushed enough answers below the pass mark to change what an automated pipeline would have shipped.

The setup is the ordinary one now running inside a great deal of production software. A judge model gets a task, an answer and a rubric, and returns a score out of 5, with 4 the mark an answer has to clear. The team changed only what sat above that in the prompt: nothing at all, a note saying the submission was a revision, or a metadata block giving the attempt number and a prior score. The prior score was drawn at random from below the pass mark, so it carried no information about the answer being graded, and the judge was not told it was random. The paper went up on arXiv on August 26. By a line the authors added to that posting it is accepted for CIKM 2026, a computing conference in Rome in November.

Across 192,000 attempted evaluations, 185,271 of which returned a usable score, the anchored prompt produced lower grades on identical text in seven of the eight judges. Claude-4.5-Sonnet moved most: the share of answers it accepted dropped by 22 percentage points. The largest standardized effect, a Cohen's d reaching 0.71, belongs to the smallest model in the set, while the largest Llama barely shifted at all. Size predicted nothing. Those evaluation counts are large because each text was graded hundreds of times over four temperature settings. The benchmark underneath them is 20 texts, spread over summarization, code review, creative writing and factual questions, and the authors state plainly that transfer to other tasks and models remains to be established.

The pull is not evenly spread across the work being graded. It was weakest on code review, where the measurement cannot be separated from zero, and strongest on factual questions and summarization. Nor does it look like sampling noise: the drift was about as large at the near-deterministic setting as at the most random one the team used.

The judge reacts to the number being there, not to its size

For one GPT-4.1 task, the team read off the probability the model assigned to each possible score digit. With no metadata in the prompt, almost all of that weight sat on the top score. With an anchor present, almost all of it sat one grade lower, and moving the anchor up and down within the tested range barely shifted it further. The response looks less like a dial than a switch. This is a probe rather than a mechanism: it is clearest in GPT-4.1, partly reproduced in the smallest Llama and absent in the two larger ones. The team also checked the obvious alternative explanation, that scores were simply bumping against the top of the scale, and found the downward shift persisted well below the ceiling.

The obvious fixes did not do what they were meant to

Two standard interventions were tried on a single GPT-4.1 scoring task. Chain-of-thought orders the model to work through each rubric criterion before committing to a score; the second is a blunt instruction to disregard prior scores and attempt numbers, ending "Metadata MUST NOT influence your judgment." Neither closed the gap between the clean prompt and the anchored one. Chain-of-thought made the gap about half again as large and cost 18% more tokens per call. The warning left the gap where it was. But it was not inert: it flattened the dose response, so the score stopped tracking the anchor's value, and on the categorical data below it recovered what the anchor cost in accuracy. What no prompt removed was the effect of the metadata being present at all.

Where it stops being a scoring quirk

The last experiment used live data: 441 messages from a campaign-compliance system, each carrying a human label of compliant, drifting or prohibited. An older classifier had gotten about half of them right. Handed the ones it had gotten wrong, GPT-4.1 corrected about a fifth with a clean prompt; with the earlier wrong label attached as metadata, anchoring blocked 48% of those corrections. On the ones the classifier had gotten right, each given a deliberately wrong label as the anchor, accuracy fell and 10.18% of the previously correct judgments flipped to the label they had been shown. A review loop built this way is least able to fix the mistakes it has already made.

None of this is the first sign that language models anchor. Published work already shows their numeric estimates being pulled toward figures they have just been shown. The literature on model judges separately documents preferences for whichever answer comes first, for longer answers, and for a model's own output. Other groups have found that two models will drop their own reading of the evidence to agree with each other. The novelty is the narrower use case: feeding a previous judge’s score back into the pipeline and testing it across eight models against the company’s own data.

The authors' recommendation is simple: keep experimentally irrelevant metadata out of the judge's context wherever the workflow allows it, and validate any mitigation on the exact model, task and domain it will run in. The warning that helped on categorical labels did nothing for numeric scores. The code, prompts, rubrics and statistical procedures are published openly, so a team running its own judge can measure the effect on its own data instead of assuming its pipeline is exempt.