A Machine-Written Disaster Report Rarely Says Where Its Numbers Came From
A situation report is not a summary. It is a dated, attributed document that a response is run from: who is affected and how many, what has been done, what is missing, and which office is saying so. That last part is what machine-written drafts leave out, almost everywhere anyone has looked. Working alone, Sachin Kumar built a 15-point checklist out of professional humanitarian reporting practice, then scored 121 professional reports published on ReliefWeb against 261 drafts written by two commercial language models that had read nothing but archived tweets from five disasters between 2017 and 2019.
The finding that survived every test is about where a number came from. The paper was posted Oct. 13, 2026, and is accepted for a workshop at the AACL-IJCNLP 2026 conference in November; it has been through workshop review rather than a journal's. It compares the drafts with the professional reports across 12 combinations of disaster and scoring model, and in all 12 the drafts supplied the report's own identifying details less often: the as-of date, the issuing organization, the update number, the stated authority. The shortfall ranged from 0.200 to 0.458 on a scale where 1.0 would mean every document in the set carried the information. Nothing else on that side of the study behaves like it. A second apparent gap, on displacement and accommodation sites, looked just as wide on Cyclone Idai and then closed on Hurricane Irma.
The reason he gives is structural rather than statistical: "a social post is not the issuing instrument for a sitrep's organisation, report number, publication date, or stated authority." A model reading a day of public posts has no access to the office that makes a situation report citable.
The figures were not in the posts
The study's strongest leg has no scoring model in it at all. A pattern matcher pulled every number out of each machine draft, out of the professional report published that day, and out of the day's complete stream of posts, then compared them exactly, with no tolerance. Of the numbers in the official report, the share that also appeared in the machine draft beside it was 11.0% on Cyclone Idai, 8.9% on Hurricane Maria and 5.4% on Hurricane Irma. Those three are the disasters with enough same-day pairs to carry the comparison; the Puebla earthquake and the Kerala floods, with eight pairs each, are kept out of the headline figure.
That is a statement about absence, and Kumar is careful about what it does not mean: "A machine figure absent from an official report is unverified, not necessarily false." What the drafts did carry came overwhelmingly from their input. Exact matching traced 85.4 to 90.1% of the numbers in the machine drafts back to that day's posts, and a hand audit of a sample found nearly all of those matches genuine rather than coincidental. Each draft had been given at most 400 of a day's posts, drawn from an archived, labeled sample of English-language tweets. So the limit sat in the source, not in the model: most official figures were never in the posts to be found, and a better model reading the same input could not have recovered them.
The scores that did not notice
Three standard ways of scoring a generated report were checked against the checklist, and none of them tracked it. ROUGE-L, which counts words a draft shares with a reference text, was not significant on any of the three events and changed direction between them. BERTScore, which compares meaning rather than wording, came out negative on all three after correcting for multiple tests, steady at about 0.45: drafts that filled more of the checklist scored worse. Part of that is length, and the association survives controlling for it. A general-purpose model asked to rate usefulness, clarity, organization and credibility did no better, and it was more generous to output from its own model family.
That matters because the shared task this paper was written for ranks submitted systems on the average of BERTScore and one other similarity score. Kumar's argument is that such a ranking "need not track operational completeness", and he says plainly that he did not test the second metric. The gap between what an evaluation measures and what its users need is not peculiar to crisis reporting: a 2021 study from a different group found that only about a quarter of ROUGE, and a sixth of BERTScore, were explained by how much information a summary and its reference actually share.
What the checklist half rests on
The checklist side of the study is much softer, and the paper says so first. Completeness here is not a practitioner's judgment. It is a language model's score of absent, partial or present on each of 15 slots, and the slots were induced by a language model from the Idai reports, most of them produced by one UN office. More than half the candidate codes that process threw up were never reviewed by a person; no second coder checked any of it, and Kumar calls the result "one defensible description of Idai reporting practice, not the only one". He re-scored 100 of the judgments by hand, blind, and agreed with the best-performing judge about nine times in ten; the other three fell below his own threshold once agreement was corrected for chance. He is also the person who approved the checklist.
Which is why one number in the paper should not be read as a result. On Cyclone Idai, under the single judge validated that way, the strongest machine setup came out fractionally above the average score of the professional reports, and the paper's own test cannot separate that margin from zero (p = .852). It reverses under the other three judges. It disappears when three short snapshot products are taken out of the professional set. And it sits inside a measurement artifact the author went looking for: the judges had been fed professional reports cut off at 20,000 characters, and re-judging a sample of the longer ones in full raised the professionals' score by nearly ten times the machine margin.
Kumar's recommendation is addressed to whoever would pick such a draft up. It is "best treated as an unverified situational signal whose figures need re-sourcing from official or assessment channels and whose provenance must come from the issuing office; it should not circulate under an agency's name." He ran no deployment study, and the five disasters are historical.