Hidden Tail Risk in In-Context Learning under Context-Query Mixture Shift

Evaluation with a prompt's task frequencies can hide errors on rare tasks, even when their examples are present. We analyze this effect in a multi-task model with independent scalar parameters bounded by $B$. A normalized finite-feature attention head induces a low-rank response matrix; among all matrices of rank at most $r$, the minimum Bayes bias is $B^2\sum_{i>r}q_{(i)}$. A shift toward uniform query weights exposes this capacity floor without changing the optimal task allocation, whereas shifts that reorder tasks can also create selection regret. We connect this bias analysis to gradient flow in an unnormalized factorized model and derive the coupled query and key gradients for the normalized head. With task coverage and separated addresses, token-preserving softmax reaches the sampling rate $O(\sigma^2/k_+)$ above an explicit temperature threshold, where $k_+$ is the target count. Synthetic experiments examine these predictions, and a separate study evaluates six pretrained checkpoints on 107,520 prompts across 80 episode seeds. In the primary 2.8B pair, Mamba is more sensitive to target repetition. Its estimated matched-to-tail accuracy gap is 5.8 percentage points (95% CI: 2.93--8.71), compared with zero for Pythia. The 95% CI crosses the prespecified practical threshold of 3 percentage points, so smaller effects remain compatible with the data. Both prespecified tests pass. Smaller pairs fail the frequency-test accuracy requirements. These findings motivate frequency-resolved evaluation in this task family. Preprint with supporting source, analysis code and data. 40 pages, 12 figures, 5 tables. Not peer reviewed. Licensing: CC BY 4.0 for author-owned text, figures and data; MIT for author-owned Python code. Third-party rights and embedded font licenses are preserved; see LICENSING.txt.