Character accuracy conceals numerical error: silent value corruption in a clinical document pipeline, and what actually prevents it

Background: Clinical decision support systems increasingly read patient data from document images. Accuracy for that stage is reported at character level, and a laboratory value is not a character. Methods: We measured the image-to-value path of a retrieval-based clinical decision support prototype under controlled image degradation, using synthetic reports carrying exact machine-readable ground truth, with the document as the experimental unit. Four experiments were run: a degradation sweep over 832 readings from one document, a resolution study of 910 readings replicated across 10 reports, a comparison of seven recognition approaches over the same 832 readings plus 1,988 readings from 20 held-out reports typeset in fonts no model had seen, and an evaluation of verification checks over those same extracted values. The 832 readings are shared between the first and third experiments by design, since the third re-reads the first experiment's document with each recognition approach. Results: At 100 dpi, character accuracy remained at 88.9% while 32.3% of laboratory values were misread. Of 188 corrupted values, 89 (47.3%) carried the same abnormality flag as the truth and so were invisible to the alerting layer. Because flags are threshold quantisations, corruption of an already-abnormal value was more often silent than corruption of a normal one (61.7% against 22.1%; risk ratio 2.80, 95% CI 1.75 to 4.47), an effect that held with the document as the unit of analysis (paired difference 43.0 percentage points, p = 0.0078). Fine-tuning the recogniser did not help: it read the document it was tuned for perfectly at low resolution, but against its own base model under identical preprocessing it lost 8.1 percentage points on held-out reports (unadjusted p = 0.031; this and the corresponding version 1 comparison do not survive correction for multiple testing). That aggregate conceals a reversal: the fine-tuned model was the most accurate reader on one half of the held-out set and the least accurate on the other. Upscaling the page before recognition was the largest single gain, improving low-resolution accuracy from 60.0% to 98.5% and the held-out accuracy of the same base model by 11.0 points (p = 0.0001). A vision-language model transcribed legible pages accurately but, on degraded pages, produced fluent values that were not printed and moved them toward the normal range. Requiring agreement between two independently built readers caught 193 of 193 corruptions on held-out data at a 19.8% review rate. Conclusions: Character-level accuracy is not a safety metric for pipelines that reduce measurements to flags, and the errors such pipelines hide fall disproportionately on patients whose values are already abnormal. In the conditions tested, verification by a second independent reader reduced undetected errors more than any improvement to the recogniser itself. All findings come from synthetic documents and bound no claim about performance in deployment.

Authors

Institutions

Publication Details

Journal
Research Square
Published
2026-11-19
DOI
https://doi.org/10.21203/rs.3.rs-11720966/v1
Primary Topic
Handwritten Text Recognition Techniques
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Character accuracy conceals numerical error: silent value corruption in a clinical document pipeline, and what actually prevents it

Mohammed Jabir Hussain Shaik
Research Square
Handwritten Text Recognition Techniques
preprint

Character accuracy conceals numerical error: silent value corruption in a clinical document pipeline, and what actually prevents it

Mohammed Jabir Hussain Shaik
preprint en

Abstract

Background: Clinical decision support systems increasingly read patient data from document images. Accuracy for that stage is reported at character level, and a laboratory value is not a character. Methods: We measured the image-to-value path of a retrieval-based clinical decision support prototype under controlled image degradation, using synthetic reports carrying exact machine-readable ground truth, with the document as the experimental unit. Four experiments were run: a degradation sweep over 832 readings from one document, a resolution study of 910 readings replicated across 10 reports, a comparison of seven recognition approaches over the same 832 readings plus 1,988 readings from 20 held-out reports typeset in fonts no model had seen, and an evaluation of verification checks over those same extracted values. The 832 readings are shared between the first and third experiments by design, since the third re-reads the first experiment's document with each recognition approach. Results: At 100 dpi, character accuracy remained at 88.9% while 32.3% of laboratory values were misread. Of 188 corrupted values, 89 (47.3%) carried the same abnormality flag as the truth and so were invisible to the alerting layer. Because flags are threshold quantisations, corruption of an already-abnormal value was more often silent than corruption of a normal one (61.7% against 22.1%; risk ratio 2.80, 95% CI 1.75 to 4.47), an effect that held with the document as the unit of analysis (paired difference 43.0 percentage points, p = 0.0078). Fine-tuning the recogniser did not help: it read the document it was tuned for perfectly at low resolution, but against its own base model under identical preprocessing it lost 8.1 percentage points on held-out reports (unadjusted p = 0.031; this and the corresponding version 1 comparison do not survive correction for multiple testing). That aggregate conceals a reversal: the fine-tuned model was the most accurate reader on one half of the held-out set and the least accurate on the other. Upscaling the page before recognition was the largest single gain, improving low-resolution accuracy from 60.0% to 98.5% and the held-out accuracy of the same base model by 11.0 points (p = 0.0001). A vision-language model transcribed legible pages accurately but, on degraded pages, produced fluent values that were not printed and moved them toward the normal range. Requiring agreement between two independently built readers caught 193 of 193 corruptions on held-out data at a 19.8% review rate. Conclusions: Character-level accuracy is not a safety metric for pipelines that reduce measurements to flags, and the errors such pipelines hide fall disproportionately on patients whose values are already abnormal. In the conditions tested, verification by a second independent reader reduced undetected errors more than any improvement to the recogniser itself. All findings come from synthetic documents and bound no claim about performance in deployment.

Research Square
Queen's University Belfast (GB)
Handwritten Text Recognition Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon