Square Root Filters for Cislunar Angles-Only Relative Orbit Estimation

Abstract Precise and robust relative orbit determination is critical for autonomous spacecraft operations in cislunar environments where nonlinear dynamics, sparse measurements, and limited ground support challenge traditional estimation approaches. This work investigates Square Root Unscented Kalman Filters (SRUKF) and their Gaussian Mixture extension (GMSRUKF) for angles-only relative orbit estimation in Near Rectilinear Halo Orbits within the Circular Restricted Three-Body Problem. While sigma-point filters like the SRUKF provide a capable baseline under nominal conditions, their single-Gaussian structure proves insufficient under degraded conditions likely in operational cislunar scenarios, including large initial uncertainty, biased initialization, and sparse measurements. To assess the robustness benefits of mixture-based filtering, this study implements a GMSRUKF with five components and no adaptive mechanisms, and shows that even this minimal extension substantially reduces failure rates and improves consistency compared to the SRUKF. The mixture-based representation enables recovery from poor initialization and intermittent measurements by capturing non-Gaussian uncertainty distributions that arise from three-body dynamics. A consider covariance formulation is also integrated to account for observer state uncertainty without joint estimation. These results demonstrate that the Gaussian Mixture square root filtering framework offers a viable path toward reliable autonomous relative navigation in cislunar space.