Optimized spin estimation in atomic sensors via Kalman filter and smoother

Spin estimation plays a crucial role in quantum metrology, particularly in systems like atomic magnetometers where atomic spin ensembles transduce magnetic-field information into measurable optical signals. However, precise spin estimation remains challenging, especially in weak polarization schemes where signals are often obscured by both classical and quantum noise. Conventional fitting methods for spin signal and variance estimation are computationally intensive, lack real-time capabilities, and exhibit sensitivity to stochastic fluctuations and data-acquisition parameters. To address these challenges, we propose and demonstrate the implementation of a Kalman filter (KF) combined with a Rauch-Tung-Striebel (RTS) smoother for enhanced spin estimation across various polarization conditions. Our comprehensive analysis of spin conditional variance estimation reveals that the KF method achieves a remarkable 4.92-dB noise suppression compared to conventional fitting methods, while the RTS smoother provides further enhancement with 6.94-dB suppression that enables quantum noise squeezing observation. These findings offer new tools for quantum noise analysis and signal recovery, offering significant potential for advancing the performance of quantum-enhanced sensors and other precision measurement technologies.