Consecutive shooting during adjoint optimization enables highly accurate estimation of unobserved state variables of high-dimensional chaotic systems

Recent advancements in machine learning have led to the development of sophisticated algorithms for automatic differentiation. These software tools simplify the implementation of the adjoint method by automating the derivation of adjoint equations, enabling researchers to apply the adjoint method to a wide range of problems without requiring extensive expertise in the underlying mathematics. One particularly significant application is the state estimation of partially observable systems. We show that for this application the performance of the adjoint method can be significantly improved by a shooting protocol in which the end of the previous segment is used as the initial condition for estimating the next segment, independent of the success of the estimate for the previous segment. The superior performance of this adjoint optimization-based consecutive shooting method is demonstrated using Lorenz-96 models with dimensions from 9 to 300.