Bayesian variable selection via collapsed importance-weighted variational inference over discrete model spaces

Bayesian variable selection under spike-and-slab priors provides interpretable posterior inclusion probabilities for individual predictors, but exploring large discrete model spaces using conventional Markov chain Monte Carlo methods can be computationally challenging. We develop a collapsed importance-weighted variational framework for Gaussian linear regression. The framework analytically integrates out the remaining model parameters and directly approximates the posterior over binary inclusion indicators using a factorised Bernoulli distribution. To improve posterior inclusion probability estimation while retaining the computational simplicity of the factorised variational family, we replace the conventional evidence lower bound with a tighter variational Rényi importance-weighted lower bound. We establish a finite-sample upper bound on posterior inclusion probability approximation errors and show that, under suitable conditions, the optimised variational inclusion probabilities converge to their exact posterior counterparts as the number of importance samples increases and the Rényi parameter decreases appropriately. We further establish that analytical marginalisation tightens the variational objective and the corresponding inclusion probability error bound, while reducing importance-weight variance and improving the asymptotic efficiency of importance-sampling post-processing. To address the challenges of estimating gradients with respect to the variational parameters when the inclusion indicators are discrete, we leverage recent advances in variance-reduced score-function estimators, together with an annealing strategy to stabilise optimisation. Simulation experiments and empirical applications demonstrate improvements in posterior inclusion probability estimation relative to standard variational inference and two existing variational approaches.

Authors

Publication Details

Journal
arXiv (Cornell University)
Published
2026-11-16
DOI
https://doi.org/10.48550/arxiv.2611.20683
Primary Topic
Markov Chains and Monte Carlo Methods
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Bayesian variable selection via collapsed importance-weighted variational inference over discrete model spaces

Minh‐Ngoc Tran, Matias Quiroz, Mai Mai
arXiv (Cornell University)
Markov Chains and Monte Carlo Methods
preprint

Bayesian variable selection via collapsed importance-weighted variational inference over discrete model spaces

Minh‐Ngoc Tran, Matias Quiroz, Mai Mai
preprint en

Abstract

Bayesian variable selection under spike-and-slab priors provides interpretable posterior inclusion probabilities for individual predictors, but exploring large discrete model spaces using conventional Markov chain Monte Carlo methods can be computationally challenging. We develop a collapsed importance-weighted variational framework for Gaussian linear regression. The framework analytically integrates out the remaining model parameters and directly approximates the posterior over binary inclusion indicators using a factorised Bernoulli distribution. To improve posterior inclusion probability estimation while retaining the computational simplicity of the factorised variational family, we replace the conventional evidence lower bound with a tighter variational Rényi importance-weighted lower bound. We establish a finite-sample upper bound on posterior inclusion probability approximation errors and show that, under suitable conditions, the optimised variational inclusion probabilities converge to their exact posterior counterparts as the number of importance samples increases and the Rényi parameter decreases appropriately. We further establish that analytical marginalisation tightens the variational objective and the corresponding inclusion probability error bound, while reducing importance-weight variance and improving the asymptotic efficiency of importance-sampling post-processing. To address the challenges of estimating gradients with respect to the variational parameters when the inclusion indicators are discrete, we leverage recent advances in variance-reduced score-function estimators, together with an annealing strategy to stabilise optimisation. Simulation experiments and empirical applications demonstrate improvements in posterior inclusion probability estimation relative to standard variational inference and two existing variational approaches.

arXiv (Cornell University)
Markov Chains and Monte Carlo Methods
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon