Deep Errors-in-Variables using a diffusion model

Abstract Errors-in-Variables is the statistical concept used to explicitly model input variable errors caused, for example, by noise. While it has long been known in statistics that not accounting for such errors can produce a substantial bias, the vast majority of deep learning models have thus far neglected Errors-in-Variables approaches. Reasons for this include a significant increase of the numerical burden and the challenge in assigning an appropriate prior in a Bayesian treatment. To date, the attempts made to use Errors-in-Variables for neural networks do not scale to deep networks or are too simplistic to enhance the prediction performance. This work shows for the first time how Bayesian deep Errors-in-Variables models can increase the prediction performance. We present a scalable variational inference scheme for Bayesian Errors-in-Variables and demonstrate a significant increase in prediction performance for the case of image classification. Concretely, we use a diffusion model as input posterior to obtain a distribution over the denoised image data. We also observe that training the diffusion model on an unnoisy surrogate dataset can suffice to achieve an improved prediction performance on noisy data.