A Tiebreak Rule for the Rare Cases Prison Risk Models Cannot Call
Two scoring systems can be built from the same prison records, tested on the same releases, and come out equally accurate. Then they can look at the same person and return different numbers. Whichever one is installed, its number goes into a file that prison psychologists, social workers and lawyers read, and that a court may later weigh when it rules on parole.
In Catalonia that is not a thought experiment: a decision support tool called RisCanvi has helped assess inmates' recidivism risk since 2009. Ashwin Singh of TU Wien and Carlos Castillo of ICREA and Universitat Pompeu Fabra in Barcelona have now put numbers on how often such disagreement actually happens, in a paper published online Oct. 11, 2026 in the proceedings of the AAAI/ACM Conference on AI, Ethics, and Society. It is, they write, the first real-world case study of its kind in recidivism risk assessment, to the best of their knowledge.
The answer came out in the reassuring direction. Singh and Castillo assembled a pool of 78 scoring models, all within 1% of the best accuracy their search could reach and all with similarly small gaps in error rates between demographic groups, then measured how often two of them, picked at random, say the same thing about the same person. It was 91% of the time, and nearly 70% of the releases they tested were scored identically by every model in the pool. In theory, it could have been much worse. The authors show that, for this group of models, agreement could fall as low as 0.53, barely better than a coin toss. The existence of many similarly accurate models, they conclude, "does not necessarily translate into severe predictive multiplicity," the term for the same person getting different predictions from equally good models.
What makes that more than arithmetic is where the remaining disagreement sits. It is concentrated in roughly 30% of cases, nearly all of them with scores landing close to the cutoff that separates one risk level from the next. The pool is not quietly uniform either, the authors report: the models weight individual features very differently and still agree, because their mistakes fall largely on the same people. The authors add one caution themselves. The agreement they measured is procedural, a property of their own training setup, and whether some other admissible model could flip a particular individual's prediction is a question they leave to future work.
Before any model, someone has to say who reoffended
The harder half of this work was not the models. It was the labels. RisCanvi is trained on whether a released person reoffended within five years, and Spanish law counts that only when someone is sentenced to prison and actually enters it. Because judicial processes are slow, a person can serve one sentence and be re-incarcerated later for a crime committed before it, so returning to prison is not the same as reoffending. Applying the rules takes trained people and time, which is why, the authors note, RisCanvi often works from training data years out of date.
So Singh and Castillo translated the legal rules into a labeling algorithm, refining it over several rounds against annotations from sentence-enforcement researchers at the Catalan justice department, who do not work with the authors and took no part in any modeling decision. They checked its labels against the official ones for every release between 2010 and 2015 before applying it to the rest. The result is a dataset of 17,542 releases from Catalan prisons between 2010 and 2019, each carrying 43 features from the inmate's last evaluation, typically six to nine months before release.
The models never saw all of it. They were trained on the 8,177 releases of 2010 to 2014 plus the 1,958 people from 2015 to 2018 already known to have reoffended, and tested on all 1,881 releases of 2019, whose five-year observation window closed in 2024. People released in the middle years who stayed out cannot be confirmed as such without waiting out their own five years, so they are left out rather than guessed at. Ten runs, differing only in a random seed, each produced its own best scoring system, which anyone who has watched two nearly identical training runs produce different models will recognize; the 78-model pool was built from those ten.
A tiebreak borrowed from the law
For the cases where the models part company, the authors propose something simple: give the person the lowest risk score any model in the pool assigns them. The principle comes from law rather than statistics. Favor rei, rule in favor of the defendant when in doubt, applies, they argue, because the law already treats these assessments as uncertain: Spanish penitentiary law asks for a probability judgment on future behavior as release approaches, so residual doubt is built into the task, and someone has to absorb it.
Their case is that it should not be the person being scored. The scoring rule behind a risk assessment is usually invisible to the inmate and to the staff evaluating them, so there is no practical way to argue that another equally valid model would have produced a less punitive number. A lowest-risk rule grants that objection automatically. "It would be hard to justify why a person receives a higher risk score just because one of several equally good models happened to be selected at random," Singh is quoted as saying in TU Wien's announcement of the work, translated here from German.
It also performs well: among the mechanisms compared, which included averaging the models, picking one at random per case, and the opposite highest-risk rule, the lowest-risk policy gave the best balanced accuracy, with comparable parity in error rates between groups for pools built to a 3% tolerance.
The cost the authors state plainly
Scoring every tie downward has a price, and the paper does not hide it: more false negatives, people scored low risk who do go on to reoffend. The authors call it a public-safety trade-off. Two things bound it. It is far fewer such misses than the system in use produces today, which on the 2019 test releases failed to flag 84% of the people who did reoffend; and the score is not the decision. In Catalonia an algorithmic assessment and a human one are made and compared, and the human one prevails. "Precisely because the data are so noisy, responsibility for the decision has to lie ultimately with human experts," Singh says in the same announcement.
None of this has been adopted. The authors did not touch the deployed system; they built their own dataset, learned their own models on it, and the lowest-risk rule is a proposal with an argument attached. Their analysis covers one operational system in one jurisdiction, and they say the results are highly contextual and may not carry to other legal systems. The authors cannot share the data, but they explain how they obtained it: through a request to the strategic planning office of Catalonia’s Department of Justice.
What the study offers beyond Catalonia is the question it found worth asking, of a kind of system the European Union's AI Act classifies as high risk. Knowing where a set of equally good models stops agreeing, the authors argue, is what makes it possible to decide who should carry the doubt. Their own ethics statement says why that is not academic: it notes reported cases where a high RisCanvi score was lowered by human evaluators, with justification, and prosecutors then cited the tool's original number in parole proceedings as grounds to oppose release.