A Way to Check Whether an AI Still Holds What It Was Told to Forget

Ask a language model something it has been made to forget and it may well refuse to answer. The refusal proves very little. A model can be trained to withhold an answer while the knowledge behind it sits untouched in its weights, ready to be pulled back out by anyone who knows how to lean on it. That gap is established ground rather than news: Xu and colleagues reported in 2025 that an unlearned model's old behavior can be restored with minimal fine-tuning, and Lee and Schoepf and their colleagues have described the same pattern of suppression standing in for deletion.

What has been missing is a cheap way to tell one case from the other. Adar Peleg, Dvir Alsheich, Tomer Ashuach and Avi Mendelson of the Technion in Haifa propose one in a peer-reviewed paper published in October 2026 in Findings of the Association for Computational Linguistics: EMNLP 2026. The detail below comes from the authors' own camera-ready version of it, posted Nov. 3, 2026. Their method never reads a word the model writes.

It works by subtraction. Take a question that names a subject, a character from Harry Potter, say, and the same question with that name swapped for an invented one. Run both through the model and subtract one internal state from the other. The shared grammar and phrasing cancel out, and what is left is whatever the model holds about that particular name. The authors take this construction, the anon-vector, from Seyitoğlu and colleagues, who used it in 2024 to build an attack that reactivates suppressed knowledge.

What is new is treating a whole batch of those leftovers as a shape. If the model holds the concept, every question about it pushes the internal state much the same way, and the differences bunch together. If the concept is simply absent, the leftover is noise and the directions scatter. Two theorems in the paper tie that picture to a standard measure of how many separate directions a set of vectors spreads across, which is what makes the idea checkable.

Invented authors stand in for everything a model cannot know

The detection half of the work is the strong half. The team ran the measure over ten instruction-tuned models from three families, Qwen, Gemma and LLaMA, pitting 96 question-and-answer pairs about Harry Potter against 100 pairs drawn from TOFU, a benchmark built entirely out of fictitious authors invented for it. Nobody has ever written about them, so no model can hold anything about them, and none answered a single question about them correctly. Every version of the measure separated the two classes in the predicted direction on all ten models and survived a correction for testing many hypotheses at once. A classifier trained with one model held out each time told known from unknown without error.

One number the authors print themselves is worth carrying: the models were not very good at Harry Potter. The best of them answered 32.3% of those questions correctly. So the known side of the comparison rests on an argument from earlier work, that this material survives large-scale pretraining, rather than on the models showing that they know it. The ideal check would be a model trained from scratch on a corpus combed clean of the subject, which almost nobody can afford. As the authors put it, "There is, therefore, no true oracle for parametric knowledge."

A second, smaller experiment asks the harder question

The second experiment is a smaller and different thing, and the paper's own abstract makes it easy to read the two as one. Here the roster is three models: LLaMA-2-7B-chat, Meta-LLaMA-3-8B-Instruct and Qwen2.5-7B-Instruct. The authors applied five published unlearning methods to each, then tried four ways of getting the erased material back: nudging the model's internal activity in a chosen direction, priming it with examples in the prompt, attaching a tuned string of adversarial text, and a statistical test of whether a passage had been in the training data.

Whether the geometry measured after unlearning tells you how much an attack will recover turns out to depend on the attack. Of the four, only in-context priming came out both significant after correction and stable when runs were held out one at a time. The adversarial suffix and the activation nudge showed a relationship across the pool as a whole, but not under the same held-out check. The membership test gave nothing usable and is dropped from the results. And the sign flips between families of attack: heavier fragmentation marks a harder target for one attack and an easier one for another. The authors offer a mechanism for that, and say in their limitations that it is read off the correlation pattern alone, with no check against what the models actually generate.

The strongest numbers in the paper come from sweeping every layer of the network separately: rank correlations of 0.85 and 0.89, both negative, for the adversarial suffix and for in-context priming, peaking one layer apart late in the network, where the model is settling on its next word. Both rest on a smaller pool of runs than the main analysis, 11 and 15, which the paper flags itself, so they are at once the most striking figures in the work and the least settled.

The authors claim less than the question invites

They call the method "a single-phase, attack-free proxy for unlearning robustness" and, in practice, "a computationally cheap, pre-attack screening step": two forward passes per question, nothing generated, cheap enough to run before anyone spends compute on a real attack, and usable where a model's output is restricted. If the geometry around a concept survives unlearning intact, they argue, an attack will probably get it back. On the methods that distort a model's internals rather than its outputs, they write that "representation-level interventions come closest to genuine deletion, and fragmentation metrics measure how far even they fall short." A screen for fragile forgetting, then, and not a certificate that anything is gone.

A reference implementation is published on GitHub, so another group can run the measure on its own models and its own forgetting procedures, which is where the second half of this will be settled.