Lineage Is Not Ability, and Twins Are Not Lineage: Dependence-Aware Differential Item Functioning Across Open Language Model Families

Item response theory lets us ask whether a benchmark item behaves differently for two groups of language models that have the same overall ability, which is the classical definition of differential item functioning (DIF). Model panels, however, are not samples of independent examinees: a base checkpoint and its instruction-tuned twin answer most items identically, and several sizes of one family share data and tokenizer. We show that this dependence breaks the standard permutation tests. On synthetic null panels shaped like ours, a modellevel label shuffle rejects a true null in 0.58 to 0.62 of cases at α = 0.05, and a leave-familyout variant that compares a held-out family against in-sample residuals rejects in 0.83 to 0.86; tests that permute whole units (twin pairs) or refit the null hold close to the nominal level. Re-auditing 29 open models (135M to 3B parameters, 12 families) on ARC-Challenge, ARC-Easy, OpenBookQA and MMLU with valid tests changes the picture we reported in an earlier draft. Family-versus-rest DIF tests that were highly significant under the naive tests (Qwen2.5, SmolLM2, Falcon3, TinyLlama) no longer survive Holm correction under any valid test, and two families (OLMo, TinyLlama) turn out to be untestable because their only evidence was a twin pair. Lineage structure is nonetheless real: ability-controlled residuals of two models from the same family but different checkpoints correlate more than residuals of models from different families (pooled Δr = 0.16, 95% CI [0.13, 0.19], permutation p ≤ 0.0001), also among base models only, and a held-out model's family is recovered with its twin removed from the reference set in 18 of 20 cases (permutation null mean 0.09). Two cautions qualify this. Raw correctness vectors recover family almost as well (0.80 versus 0.90), so ability control is not what makes attribution work, and cross-family pairs that share a tokenizer are about as similar as same-family pairs, so part of what looks like lineage may be tokenization under log-likelihood scoring. A paired within-twin test also overturns our earlier null for instruction tuning: tuning appears to leave a modest item-level signature (pooled exact p = 0.002; p = 0.004 after removing smooth person misfit), although part of it reflects instruct models fitting the IRT model less well. Rankings are stable when high-DIF items are removed. Finally, an IRT-residual contamination detector beats label-only baselines in controlled LoRA injections but is dominated by Min-K% Prob whenever token probabilities are available, so we do not recommend it. We release the tests as a small open tool.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-17
DOI
https://doi.org/10.5281/zenodo.24326149
Primary Topic
Psychometric Methodologies and Testing
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Lineage Is Not Ability, and Twins Are Not Lineage: Dependence-Aware Differential Item Functioning Across Open Language Model Families

Sharven Rane
Zenodo (CERN European Organization for Nuclear Research)
Psychometric Methodologies and Testing
preprint

Lineage Is Not Ability, and Twins Are Not Lineage: Dependence-Aware Differential Item Functioning Across Open Language Model Families

Sharven Rane
preprint en

Abstract

Item response theory lets us ask whether a benchmark item behaves differently for two groups of language models that have the same overall ability, which is the classical definition of differential item functioning (DIF). Model panels, however, are not samples of independent examinees: a base checkpoint and its instruction-tuned twin answer most items identically, and several sizes of one family share data and tokenizer. We show that this dependence breaks the standard permutation tests. On synthetic null panels shaped like ours, a modellevel label shuffle rejects a true null in 0.58 to 0.62 of cases at α = 0.05, and a leave-familyout variant that compares a held-out family against in-sample residuals rejects in 0.83 to 0.86; tests that permute whole units (twin pairs) or refit the null hold close to the nominal level. Re-auditing 29 open models (135M to 3B parameters, 12 families) on ARC-Challenge, ARC-Easy, OpenBookQA and MMLU with valid tests changes the picture we reported in an earlier draft. Family-versus-rest DIF tests that were highly significant under the naive tests (Qwen2.5, SmolLM2, Falcon3, TinyLlama) no longer survive Holm correction under any valid test, and two families (OLMo, TinyLlama) turn out to be untestable because their only evidence was a twin pair. Lineage structure is nonetheless real: ability-controlled residuals of two models from the same family but different checkpoints correlate more than residuals of models from different families (pooled Δr = 0.16, 95% CI [0.13, 0.19], permutation p ≤ 0.0001), also among base models only, and a held-out model's family is recovered with its twin removed from the reference set in 18 of 20 cases (permutation null mean 0.09). Two cautions qualify this. Raw correctness vectors recover family almost as well (0.80 versus 0.90), so ability control is not what makes attribution work, and cross-family pairs that share a tokenizer are about as similar as same-family pairs, so part of what looks like lineage may be tokenization under log-likelihood scoring. A paired within-twin test also overturns our earlier null for instruction tuning: tuning appears to leave a modest item-level signature (pooled exact p = 0.002; p = 0.004 after removing smooth person misfit), although part of it reflects instruct models fitting the IRT model less well. Rankings are stable when high-DIF items are removed. Finally, an IRT-residual contamination detector beats label-only baselines in controlled LoRA injections but is dominated by Min-K% Prob whenever token probabilities are available, so we do not recommend it. We release the tests as a small open tool.

Zenodo (CERN European Organization for Nuclear Research)
Psychometric Methodologies and Testing
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon