Training the system around a frozen language model: a clinician's lessons and code checks on synthetic intensive care cases — a 2 × 2 factorial study with a measurement of expert reliability

Aim. To test whether the decision of a local language model whose weights do not change becomes clinically more appropriate if it is given a memory of my lessons and code checks — and what each of them contributes; and, at the same time, to measure myself as the reader who judges these decisions. Methods. The system is built on the local model Gemma 4 26B A4B in the role of the intensivist on duty: the model reads a synthetic case, assesses the data, and writes one rescue decision with its rationale and an order sheet. Around the unchanged model the system learns in two ways: memory — 26 lessons written from my verdicts (I am an intensive care physician); code — checks that reject answers without a decision, carry a lesson through to action and put the order sheet in order. I compared four variants of one system — with nothing, with lessons, with checks, with both — on 10 new cases in three languages, three replicates each: 120 answers. I rated each answer twice — immediately, knowing the variant, and two weeks later blind, in random order — as appropriate, partially appropriate or inappropriate. The comparison is based on the blind reading; the unit of analysis is the case. Results. Between the two readings I agreed with myself in 54% of verdicts (κ = 0.24). In the blind reading the worst variant was lessons alone: not a single appropriate decision and 14 inappropriate out of 30; the full system — 6 appropriate and 6 inappropriate; the bare model and checks alone — in between. The largest difference was the system versus lessons alone (better in 8 cases of 10; nominal p = 0.02, 0.14 after correction for six comparisons); the system versus the bare model — not better. The logs revealed the mechanism: the model always cites a retrieved lesson but does not carry it out without checks (mandatory lesson components missing from the first answer appeared in 0 of 9 answers without checks versus 7 of 9 with checks); the checks also carry wrongly retrieved lessons through to action. The reader kept learning and grew tired — this is visible in his notes and in the length of his comments, but the data show no shift of ratings under fatigue; the least stable step of the scale was the top one. Conclusions. Without checks, memory is the worst variant; with checks it is no worse than the bare model and gives the fewest inappropriate decisions; the system did not become better than the bare model. A lesson transfers to its own error, and only when the system has recognised it. The reader is part of the system and must be measured just as the model is.