Correcting Autodifferentiation in Neural ODE Training

Abstract. Does the use of autodifferentiation yield reasonable updates for deep neural networks (DNNs)? Specifically, when DNNs are designed to adhere to neural ODE architectures, can we trust the gradients provided by autodifferentiation? Through mathematical analysis and numerical evidence, we demonstrate that when neural networks employ high-order methods, such as linear multistep methods or explicit Runge–Kutta Methods (ERK), to approximate the underlying ODE flows, brute-force autodifferentiation often introduces artificial oscillations in the gradients that prevent convergence. In the case of leapfrog and 2-stage ERK, we propose simple postprocessing techniques that effectively eliminate these oscillations, correct the gradient computation, and thus return the accurate updates.