Context Curation Is Not a Free Lunch: An Empirical Test of Inference-Time Robustness for Tabular Foundation Models Under Distribution Shift

Tabular foundation models such as TabPFN introduced a different deployment paradigm: instead of fitting parameters to a training set, the model performs in-context learning, conditioning a pretrained transformer on a labeled context set at inference time, with no gradient-based training. This training-free deployment is the paradigm's central advantage, but it is threatened by distribution shift: deployment drift degrades predictive performance, and existing remedies retrain the pretraining prior, reintroducing the cost the paradigm was meant to avoid. Because the context set is already a free parameter at inference time, we ask a question that recent concurrent work has approached only indirectly: can selecting which training rows populate the context, using only unlabeled target features, substitute for retraining as a lightweight defense against distribution shift? Unlike concurrent context-selection work motivated primarily by inference efficiency (CRUMB) or cold-start, low-label performance (LUCoS), we treat context curation as a shift-robustness mechanism in its own right and test it directly against a matched retraining baseline across multiple shift types, severities, and a real-world drift benchmark—a comparison neither concurrent work performs. We evaluated five context-selection strategies against TabPFN on five public tabular datasets under three constructed shift types and four severities, with three random seeds and three context sizes, yielding 540 matched evaluation observations for the primary context-selection comparisons, plus one real-world concept-drift benchmark. Contrary to our hypothesis, the best-performing strategy, density-reweighted context selection, did not reliably outperform random selection (p = 0.068), did not outperform an equivalently-informed retraining baseline, performed significantly worse than a tuned retraining baseline (p = 0.008), and underperformed random selection on the real-world drift benchmark. A shift-detectability diagnostic and six candidate moderators showed little reliable relationship with the method's benefit. We did, however, find a robust negative result: nearest-to-target and class-balanced context selection significantly underperformed random selection across the evaluation grid, with paired mean accuracy differences of −3.67 and −3.31 percentage points, respectively. We report this as a robust cautionary result: target-awareness in context selection is not sufficient to improve robustness, and a plausible heuristic can actively harm performance. We release the full experimental pipeline for reproducibility.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-13
DOI
https://doi.org/10.5281/zenodo.24198180
Primary Topic
Data Stream Mining Techniques
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Context Curation Is Not a Free Lunch: An Empirical Test of Inference-Time Robustness for Tabular Foundation Models Under Distribution Shift

Mansoob-e-Zahra
1 links
Zenodo (CERN European Organization for Nuclear Research)
Data Stream Mining Techniques
preprint

Context Curation Is Not a Free Lunch: An Empirical Test of Inference-Time Robustness for Tabular Foundation Models Under Distribution Shift

Mansoob-e-Zahra
preprint en

Abstract

Tabular foundation models such as TabPFN introduced a different deployment paradigm: instead of fitting parameters to a training set, the model performs in-context learning, conditioning a pretrained transformer on a labeled context set at inference time, with no gradient-based training. This training-free deployment is the paradigm's central advantage, but it is threatened by distribution shift: deployment drift degrades predictive performance, and existing remedies retrain the pretraining prior, reintroducing the cost the paradigm was meant to avoid. Because the context set is already a free parameter at inference time, we ask a question that recent concurrent work has approached only indirectly: can selecting which training rows populate the context, using only unlabeled target features, substitute for retraining as a lightweight defense against distribution shift? Unlike concurrent context-selection work motivated primarily by inference efficiency (CRUMB) or cold-start, low-label performance (LUCoS), we treat context curation as a shift-robustness mechanism in its own right and test it directly against a matched retraining baseline across multiple shift types, severities, and a real-world drift benchmark—a comparison neither concurrent work performs. We evaluated five context-selection strategies against TabPFN on five public tabular datasets under three constructed shift types and four severities, with three random seeds and three context sizes, yielding 540 matched evaluation observations for the primary context-selection comparisons, plus one real-world concept-drift benchmark. Contrary to our hypothesis, the best-performing strategy, density-reweighted context selection, did not reliably outperform random selection (p = 0.068), did not outperform an equivalently-informed retraining baseline, performed significantly worse than a tuned retraining baseline (p = 0.008), and underperformed random selection on the real-world drift benchmark. A shift-detectability diagnostic and six candidate moderators showed little reliable relationship with the method's benefit. We did, however, find a robust negative result: nearest-to-target and class-balanced context selection significantly underperformed random selection across the evaluation grid, with paired mean accuracy differences of −3.67 and −3.31 percentage points, respectively. We report this as a robust cautionary result: target-awareness in context selection is not sufficient to improve robustness, and a plausible heuristic can actively harm performance. We release the full experimental pipeline for reproducibility.

Zenodo (CERN European Organization for Nuclear Research)
Pakistan Institute of Engineering and Applied Sciences (PK)
Data Stream Mining Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon