Deterministic and probabilistic rounding error analysis of neural networks in floating-point arithmetic

Abstract The use of artificial neural networks is now becoming widespread across a wide variety of tasks. In this context of very rapid development issues related to the storage and computational performance of these models emerge, since networks are sometimes very deep and comprise up to trillions of parameters. For all these reasons the use of reduced precision is increasingly being considered, although, until now, its accuracy and robustness had been approached mostly from a practical standpoint or verified by software. The aim of this work is to provide formal tools to better understand, explain and predict the accuracy and stability of neural networks when using floating-point arithmetic. To this end, we apply a rounding error analysis based on existing tools in numerical linear algebra to obtain both forward and backward error bounds. This includes both deterministic worst-case bounds as well as probabilistic bounds that are sharper on average. Since the exact backward error is not directly computable the backward error bounds are validated experimentally through a linearized backward error. These bounds both ensure the proper functioning of neural networks once trained and provide recommendations on architectures and training methods to enhance the robustness of neural networks.