Infinitely deep Bayesian neural network with signature transform

This work introduces a novel partially infinitely deep Bayesian neural network, in particular, where the weights in each layer are governed by differential equations driven by the signature of Brownian motion. By leveraging the advantages of the randomness compression for the stochastic processes through signature transforms, our architecture is designed to overcome the limitations of current continuous deep Bayesian neural networks under stochasticity, particularly in terms of reliability and robustness. Additionally, we present a comprehensive mathematical framework that integrates the signature transform of stochasticity into the weight evolution of the Bayesian neural network. To approximate the true posterior, we adopt the approximate Bayesian computation method. Finally, empirical results across image classification tasks (CIFAR-10, CIFAR-10C) demonstrate that our model outperforms existing neural stochastic differential equation models in reliability and robustness while maintaining the memory-efficient training and accuracy of neural stochastic differential equations. • We introduce the signature transform to characterize the stochasticity in continuous-time neural mod- els, extending the current framework of continuous Bayesian neural network and its descendants. • We present a comprehensive mathematical framework that integrates the signature transform of stochas- ticity into the weight evolution of infinitely deep Bayesian neural networks. • We demonstrate that our rough differential equation models outperform stochastic networks in terms of reliability and robustness while still capturing the inherent stochasticity, highlighting that a more efficient application of deep signature methods is possible.