Conflict or Ignorance in the Rashomon Set? Certified Typed Verdicts for Variable Importance

When many models are almost equally accurate, they can disagree about which variables matter. Reports of this disagreement, as a range or a distribution of importance values, rarely say whether it would disappear with more data or would remain with unlimited data, and the two situations call for different actions. We compare the range of a variable’s importance over the population ε-Rashomon set, read as a partially identified quantity, with an importance threshold, and return for each variable one of four verdicts: IMPORTANT and IRRELEVANT (every ε-good model is on the same side of the threshold), STRUCTURAL (ε-good models lie on both sides, a property of the population, the model class and the tolerance that does not change with the sample size) and IGNORANCE (an abstention, with a projection of the sample size that would settle it). The probability of certifying at least one wrong verdict is controlled at level α asymptotically. For linear regression the verdicts have a closed form, and, for coefficients on fixed (population) scales under a correctly specified homoskedastic model, collinearity inflates a coefficient’s identified half-width and its standard deviation by the same factor. For finite model classes, such as all depth-two trees on a split grid, we build outer and inner confidence sets for the Rashomon set from loss differences to a reference fixed on a held-out split, a paired permutation estimator with independent units, and a studentized multiplier bootstrap. In simulations, including designs with an importance endpoint within 0.005 of the threshold or a model about 0.001 from the tolerance boundary, one wrong verdict was certified in more than 2,400 runs, and the empirical family-wise error stayed below α in every setting; bounding raw losses instead of loss differences lowered the share of replicates certifying STRUCTURAL at n = 20,000 from 0.73–0.83 to 0.33–0.67, and adapted general concentration bounds of Fisher et al. (2019), with an empirical loss bound, certified one linear verdict in six at n = 100,000. In a diabetes-screening illustration (BRFSS 2015), the best single tree assigns zero importance to heart disease and walking difficulty, while the procedure certifies both as STRUCTURAL at a one-point error tolerance in two of three sample splits, and reports IGNORANCE, never a contradictory verdict, in the third. Code is released as the Python package typedvi.

Authors

Publication Details

Journal
Preprints.org
Published
2026-11-13
DOI
https://doi.org/10.20944/preprints202611.1456.v1
Primary Topic
Statistical Methods and Inference
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Conflict or Ignorance in the Rashomon Set? Certified Typed Verdicts for Variable Importance

Maikel Yelandi Leyva Vázquez, Alexis Matheu Pérez, Bastián Gutiérrez
Preprints.org
Statistical Methods and Inference
preprint

Conflict or Ignorance in the Rashomon Set? Certified Typed Verdicts for Variable Importance

Maikel Yelandi Leyva Vázquez, Alexis Matheu Pérez, Bastián Gutiérrez
preprint en

Abstract

When many models are almost equally accurate, they can disagree about which variables matter. Reports of this disagreement, as a range or a distribution of importance values, rarely say whether it would disappear with more data or would remain with unlimited data, and the two situations call for different actions. We compare the range of a variable’s importance over the population ε-Rashomon set, read as a partially identified quantity, with an importance threshold, and return for each variable one of four verdicts: IMPORTANT and IRRELEVANT (every ε-good model is on the same side of the threshold), STRUCTURAL (ε-good models lie on both sides, a property of the population, the model class and the tolerance that does not change with the sample size) and IGNORANCE (an abstention, with a projection of the sample size that would settle it). The probability of certifying at least one wrong verdict is controlled at level α asymptotically. For linear regression the verdicts have a closed form, and, for coefficients on fixed (population) scales under a correctly specified homoskedastic model, collinearity inflates a coefficient’s identified half-width and its standard deviation by the same factor. For finite model classes, such as all depth-two trees on a split grid, we build outer and inner confidence sets for the Rashomon set from loss differences to a reference fixed on a held-out split, a paired permutation estimator with independent units, and a studentized multiplier bootstrap. In simulations, including designs with an importance endpoint within 0.005 of the threshold or a model about 0.001 from the tolerance boundary, one wrong verdict was certified in more than 2,400 runs, and the empirical family-wise error stayed below α in every setting; bounding raw losses instead of loss differences lowered the share of replicates certifying STRUCTURAL at n = 20,000 from 0.73–0.83 to 0.33–0.67, and adapted general concentration bounds of Fisher et al. (2019), with an empirical loss bound, certified one linear verdict in six at n = 100,000. In a diabetes-screening illustration (BRFSS 2015), the best single tree assigns zero importance to heart disease and walking difficulty, while the procedure certifies both as STRUCTURAL at a one-point error tolerance in two of three sample splits, and reports IGNORANCE, never a contradictory verdict, in the third. Code is released as the Python package typedvi.

Preprints.org
Statistical Methods and Inference
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon