Specificity Collapse and Calibration Drift under External Schema Shift: An Empirical Case Study of Tabular-to-Text Transformers and Large Language Models in Clinical Prediction

Background. Clinical prediction models often degrade under external deployment, but the resulting failure patterns may depend on representation and inference paradigm. We studied this question in a small-scale clinical prediction setting using a tabular-to-text Transformer, a frozen large language model (LLM), and conventional tabular baselines. Method. We trained a custom GPT-style tabular-to-text Transformer on the UCI Heart Disease dataset (n=303; 5-fold stratified cross-validation) and compared it with eight ML baselines and a locally deployed LLM (Qwen3.5-2B) under 0-, 1-, 3-, 5-, and 10-shot prompting. External evaluation used a 918-patient Kaggle heart disease dataset with only 11 corresponding predictors; two UCI features (ca, thal) were introduced as alignment variables and set to 0 to match the training schema. The Transformer used a Chinese tabular-to-text pipeline, whereas the LLM used an English natural-language rendering with semantically explicit missing-value wording. Primary metrics were AUC, sensitivity, and specificity. Results. The Transformer achieved internal AUC 0.762 ± 0.070 and specificity 0.811 ± 0.084, but on the aligned external cohort its mean AUC fell to 0.624 ± 0.033 and specificity to 0.656 ± 0.279 (Δspecificity = −0.156). Attention comparisons on matched inputs were consistent with disrupted routing around the thal token, but should be interpreted as mechanistic probes rather than definitive causal proof. The LLM showed strong zero-shot positive bias (specificity 0.311 ± 0.043), whereas 5-shot prompting yielded the most stable profile (internal AUC 0.755 ± 0.030; external AUC 0.739; external specificity 0.627). In an ablation that altered the aligned external encoding, mean imputation of ca/thal degraded Transformer performance relative to zero-padding (ΔAUC = −0.125), suggesting sensitivity to tokenization-level familiarity. Conclusions. In this case study, the tabular-to-text Transformer and the prompted LLM exhibited different external failure patterns: the former showed a marked specificity drop under schema alignment shift, whereas the latter mainly showed calibration sensitivity to prompt construction. Because the comparison also includes differences in representation language, missing-value semantics, and external schema alignment, the findings should be viewed as hypothesis-generating evidence for clinical AI reliability rather than a definitive architecture-level ranking.