Active machine learning using MCMC Bayesian neural networks

Background Standard neural networks require large labelled datasets for strong generalization, which is often costly to annotate. Active learning (AL) reduces annotation costs by selecting the most informative samples for labelling based on model uncertainty, and Bayesian methods are a natural source of such uncertainty estimates. Approximate Bayesian methods such as Monte Carlo Dropout (MC Dropout) are scalable but can introduce systematic approximation errors in low-data regimes, whereas Markov Chain Monte Carlo (MCMC) provides higher-fidelity posterior estimates but is typically too computationally expensive for deep learning. We note that MCMC and AL are potentially complementary, since AL is designed to keep the labelled set small, which is exactly the regime in which MCMC is most tractable. Methods We investigate whether a small Bayesian neural network trained with MCMC outperforms a larger network using MC Dropout, when both use the same query function, Power Bayesian Active Learning by Disagreement (PowerBALD), across eight tabular classification datasets, an active learning setting with a 100-sample annotation budget, and matched supervised-learning baselines at 100 samples and at full training-set size. Results Within the 100-sample annotation budget, the small MCMC-based model achieved higher balanced accuracy than the MC Dropout model in 6 of 8 active learning tasks, and in 4 of 8 tasks in the matched 100-sample supervised setting, with one additional tie. The active learning procedure also matched or exceeded the performance of training on the full dataset in 5 of 8 tasks while using only a small fraction of the annotations. Conclusions These results suggest that MCMC is a viable and, in this constrained-budget setting, often preferable alternative to approximate Bayesian inference for active learning, though we do not claim that it is a generally superior inference method outside this low-annotation regime.