Broken Input Looks Exactly Like Bad Reasoning
When a language model pipeline produces a wrong judgment, the field's default diagnosis is that the model reasoned badly. This paper documents four cases from a production legal citation verification system where that diagnosis would have been wrong. In each, the model reasoned correctly and consistently over text it had never been shown in complete or accurate form. A text-handling defect upstream of the model produced the wrong output — a hyphen, a character cap, a sentence boundary, and a privacy filter. The distinction matters because the two failure modes require opposite fixes and are observationally identical from the output side. The fourth case is a distinct and more important sub-class: the corruption was not a defect at all. A privacy-redaction pass working exactly as designed destroyed the evidence it existed to make safe to analyse, and the damage propagated into a second subsystem where it silently disabled a human correction. All four are named against public opinions, with the specific text, so the claims can be checked.
Authors
- Benjamin Schulz
Institutions
- Computer Algorithms for Medicine (AT)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-08-24
- DOI
- https://doi.org/10.5281/zenodo.22072931
- Primary Topic
- Topic Modeling
- Type
- preprint