The language model on a drifting speech decoder

A speech neuroprosthesis decodes phonemes from motor cortex, and a language model then chooses among the decoder's 100 best sentences. Published evaluations of that rescoring stage use random splits over sentences and never ask what it does on sessions recorded after calibration. On the Card et al. (2024) recordings from participant T15 (45 sessions over 20 months) and the Willett et al. (2023) recordings from participant T12, with Cinà's cached candidate lists and rescorer scores, we evaluate rescoring, a correct-or-keep gate fit on early sessions, and a probe on the rescorer's activations, all held out by session. On a calibrated decoder, rescoring is churn with a small net: on Cinà's random test split Qwen2.5-7B fixes 85 words and breaks 63, and on T12, held out by session, OPT-6.7b nets zero. A gate on output features halves the number of sentences the language model makes worse on T15, at an unchanged word error rate point estimate, and replicates on T12 for OPT only. The activations do not help: a probe on them leaves the gate 0.11 points worse (95% interval +0.04 to +0.20). A decoder retrained on the first 22 sessions and used without recalibration has three times the released checkpoint's phoneme error rate one week after its last training session, and 74 to 81% after a year; the released checkpoint run through the same frozen input layer loses about three quarters of that gap, so most of the drift sits in the input layer. In the first month of drift the rescorer removes a larger share of errors and the gate vetoes helpful changes. After three months the lists no longer contain the sentence, so nothing that chooses among them can recover it. One account fits these results: the pipeline weights the second-pass language model the same for every sentence, while the decoder's reliability varies between sentences and falls under drift. The account predicts that weighting the language model by the decoder's own confidence would help, and the committed candidate lists are enough to test the sentence-level version. Predictions were committed before each analysis, and timestamped from the gate step on.