A Benchmark Scored a Reply That Never Answered the Question as Correct

A vision-language model is shown a photograph of an industrial part and asked whether it is defective, with two options: (A) Yes, (B) No. What comes back is a description. Ahmad Jarjis Hasan, an independent researcher, and Jonathan Lwowski of Nebius open their audit of benchmark scoring code with that exchange: "'The object is a nail.' The model never answered the question." MMAD, the industrial-inspection benchmark that asked, scored the reply correct.

That is the paper's own summary of the problem: "The grade came from the grader, not the model." Their audit took the released scoring code of eight peer-reviewed benchmarks for models that read images and text together. They ran it over stored answers from five open vision-language models, each small enough for a single 16 GB graphics card, and measured what the code alone was worth. Changing only the rule a scorer applies when it cannot find an option letter moved reported accuracy by as much as 22 points on answers that did not change by a single byte. The work reached arXiv on Nov. 2, 2026.

That last clause is where the force of the result sits. The answers were generated once with sampling switched off, so the text is fixed, then cached and graded again and again by different rules over the same bytes. For each benchmark, the two cloned the official repository at a pinned commit and called its own scoring function. As the paper puts it: "we do not reimplement the scorer." Whatever separates two scores of the same text, it is not the model.

Where the biggest number comes from

That 22 is the extreme of the set, and the authors are careful about where it lives. It belongs to MMAD, a benchmark of yes-or-no industrial inspection questions, and to the only two models in the audit that ramble instead of answering with a letter, Pixtral-12B and Llama-3.2-11B. It compares two choices available to any benchmark when the letter is missing: snap the reply to whichever option it most resembles, or mark it wrong. Elsewhere the effect is much smaller. The paper's own limitations section puts the certified magnitudes at 1 to 5 points, reaching 8 to 10 on the models that ramble. The biggest artifact landing on the simplest benchmark is mechanical: a two-option snap concentrates its manufactured verdicts on the answer that is usually right, while a random fallback spreads them thin.

What that snap looks like in practice is a few lines of Python. MMAD's released evaluation harness looks for a bare option letter with a regular expression, and when it finds none it hands the whole reply to difflib's get_close_matches, a standard routine for picking the nearest string, with the similarity cutoff set to 0.0. At zero there is no threshold at all: every string has a closest option, so every reply gets a verdict, including one about a nail. The repository ships 13 such harnesses, and the function that totals their scores is spelled caculate_accuracy_mmad, a misspelling that has survived in released code nobody had reason to open.

Rambling is not the bug

The temptation is to blame the models for not following instructions, and two control benchmarks in the audit close that door. MathVision draws the highest rate of unreadable replies of anything in the set, and its artifact is exactly zero, because its scorer marks what it cannot read as wrong rather than guessing. MathVista puts a language model in front of the same snap to extract the answer first; scored raw, it shows a spurious 10-point artifact, and scored faithfully, that collapses to under a point. "Rambling is not the bug; the fallback is," the authors write. Six of the eight benchmarks they audited manufacture a verdict when the letter is missing; the other two decline to.

There is also no safe rule to switch to. Checked against a human reading of what each model actually said, the snap scores items correct that the model never answered, while the letter-only rule discards answers that were stated in words. "There is no neutral default," the authors write.

On the models that answer with a clean option letter, the same rule change moves accuracy by exactly zero. That is why this has stayed invisible: it never touches the well-behaved systems whose scores get argued over, only the smaller open models a hospital or a factory can run on its own hardware. It also keeps the finding narrower than it sounds. Under a heading titled "What we do not claim," the authors write: "Holding each benchmark's parser fixed and varying only the fallback, we find no robust leaderboard reordering." Where the parser itself is the variable, the picture changes. Accepting a near-miss answer format on ScienceQA lifts Llama-3.2-11B from last place to fourth, which they report without building on it.

Nor does the problem retire as models improve. It moves. On MMAD, a reasoning version of Qwen3-VL-8B never once failed to produce a clean letter, so the fallback fired zero times and the artifact this paper measures disappeared. Its grade still swung 9.5 points, because its visible reasoning restates the option list before it answers, and 11 of MMAD's 13 harnesses keep the last letter the regular expression finds, while two keep the first. The benchmark ships both numbers and reports neither choice. "Scoring rules are revised in practice, just not in print," the authors write.

The remedy is almost free

What Hasan and Lwowski ask for is not new machinery. Report the non-compliance rate, the share of answers on which the scorer had to fall back on a guess, beside the accuracy. Publish accuracy under two rules instead of one, because the spread between them is the sensitivity a single figure hides. And adjudicate where it matters. Their blinded protocol shows a reader the question, the options and the model's reply while withholding the right answer and every scorer's verdict, and it cost about one person-day per benchmark. The labels, the code, the judge and a one-command containerized reproduction are released under MIT and Creative Commons terms, so that "any benchmark author can audit what their grader credits."

The audit is accepted at WACV 2027, in the conference's evaluations and datasets track, and the camera-ready text reached arXiv on the day that version was due; the proceedings do not exist until the conference meets Jan. 4 to 8, 2027, so this is work in press rather than published. One of its two human annotators was an author, as the paper discloses, and the second was naive to what was being tested. Its closing line asks for the scrutiny the field already gives its data: "the dataset is not the only thing worth reviewing."