Checking AI-generated clinical documentation against its source record: twelve sealed evaluations of a deterministic checker and an exploratory language-model second reader

File of record: When_the_record_is_silent_v1.6.3.pdf. The superseded 1.6.2 file also appears on this version because an upload during a Zenodo outage on 12 October 2026 attached it and published files cannot be removed; it is identical to the 1.6.2 record and carries the two remnants 1.6.3 corrects. Background. AI-generated clinical documentation may preserve much of its source wording while altering factual content, temporal status or evidential qualification. Deterministic checking offers reproducible comparison against the source, but its performance and limitations on free-text clinical documentation are uncertain. This study evaluated a rule-based checker across sequential sealed synthetic corpora and explored whether a local language model could add detection when used as a second-stage reader. Methods. A synthetic pre-operative assessment record, developed through three versions, provided the source. A commercial AI documentation system generated notes from the record text alone. Twelve sequential sealed evaluation corpora, each of 100 notes written by language-model writers given the record and, except for one writer in each of the last six corpora, fifteen distortion definitions, were scored once against checker versions frozen before each corpus was written. The checker applies six primary labels and eleven supplementary deterministic checks. Outcomes were faithful lines flagged (false warnings), planted distortions missed, misses within a pre-specified consequential category, and binary classification accuracy. After the eleventh corpus, development was concluded; a masked two-pass author adjudication of 96 disputed and sampled lines against a written interpretation standard, and a bounded maintenance release built from its findings with those lines as regression tests, were then scored once on a twelfth corpus, with the retained release scored on the same notes. The seventh to twelfth corpora included writers of three model classes and one writer operating under a plain clinician's brief without distortion categories. An exploratory experiment, specified before any verdict was read, compared the checker alone, a local 35-billion-parameter model alone, and two serial configurations, on the seventh to twelfth corpora. Results. On the twelfth corpus, written after development was concluded, the maintenance release achieved 91.3 per cent binary accuracy, detecting 79.5 per cent of planted distortions and 93.6 per cent of pre-specified consequential distortions, with 4.3 per cent false warnings; the retained reference release scored 89.4, 79.2, 94.9 and 6.8 per cent on the same notes, the one consequential difference a single reversal the maintenance release excuses, recorded as a defect. The six labels alone scored 80.2, 31.3, 56.4 and 2.0 per cent. The language model confined to the lines the maintenance release passed, with omission-only verdicts set aside, raised accuracy to 93.0 per cent and consequential detection to 100 per cent (0 of 78 missed), with 5.1 per cent false warnings; without the filter 92.1, 100 and 6.6, the filter this time setting aside three catches as well as 18 false warnings. Masked adjudication of 96 lines found the dropped self-report check right on 24 of 24 (a qualification lost, consequential on the functional-capacity and exertional-symptom lines), the status-verb check wrong on 11 of 24 fully faithful lines, and 22 of 24 misses real, 17 of them consequential. On the eleventh corpus the six-label configuration achieved 82.3 per cent binary accuracy, detecting 40.9 per cent of planted distortions and 67.7 per cent of pre-specified consequential distortions, with 3.4 per cent false warnings; adding the supplementary checks raised these to 87.5, 79.9 and 96.9 per cent, with 9.8 per cent false warnings. Across the twelve corpora the planted-distortion miss rate fell from 32.0 per cent in the first evaluation to between 16.9 and 20.8 per cent in the last four, and the pre-specified consequential miss rate from 23.3 per cent in the second to between 2.6 and 6.4 per cent in the last four. Conclusions. Deterministic source checking identified most planted distortions but remained limited by proposition-level changes that line alignment and word overlap do not reliably identify. A serial language-model reader added detection at the cost of more false warnings. Findings are restricted to synthetic material from one clinical record, with author-dependent reference assessment and writers from one model family. Independent clinical validation, and evaluation of effects on clinicians, remain outstanding. Version 1.6.3 (12 October 2026): as 1.6.2, with a duplicated fragment in the abstract and the record range in Figure 1's caption corrected; no change to any figure. Version 1.6.2 (12 October 2026): as 1.6, with text that still counted eleven corpora corrected throughout (abstract, 2.3, 2.5, 3.7, 4.2) and a garbled sentence in 3.4 repaired; no change to any figure. Version 1.6.1 corrected the abstract only. Version 1.6 (12 October 2026): the masked two-pass author adjudication of 96 lines against a written interpretation standard, reported as a separate analysis; the bounded maintenance release built from it (fingerprint 7b2a3e0c0f1f), scored once on a twelfth sealed corpus with the retained release on the same notes; the second reader re-run on that corpus. Version 1.5 (11 October 2026): reporting corrections after independent review (pooled false warnings 540 not 539; net model false warnings 39 not 40; the unchanged-rules range stated) and the stopping decision that concludes the optimisation programme at the eleventh corpus; no new figures. Version 1.4 adds an eleventh sealed corpus scoring a filter on the second reader's verdicts (omission-only verdicts set aside), with no rule change; version 1.3 adds a tenth sealed corpus written under a brief that settles the self-report marker disagreement on the rule's side, with no rule change; version 1.2 adds a ninth sealed corpus scoring four fixes to the checker's own false-warning classes found in the eighth; version 1.1 adds an eighth sealed corpus written after two post-development rule fixes, with the page, tables and second-reader experiment updated to it; version 1.0 reported seven corpora. The live checker, its figures and its evaluation manifests are public at https://fellowship.narrativelogic.co.uk/#record-silent. The vendor of the commercial AI documentation system is not named.