Spatio-temporal memory guided factor graph optimization for robust 3D LiDAR multi-object tracking in autonomous driving

Reliable three-dimensional (3D) multi-object tracking (MOT) is essential for dynamic scene understanding in autonomous driving and intelligent traffic monitoring. Existing tracking-by-detection methods commonly rely on hand-crafted motion models, which limits their robustness under irregular object motion, temporary occlusions, and noisy detections. To address these limitations, we introduce a spatio-temporal memory guided factor graph optimization framework for 3D LiDAR MOT. Specifically, an LSTM-based spatio-temporal model is employed to encode historical trajectories and generate uncertainty-aware motion predictions, which are explicitly incorporated into data association to handle prediction ambiguity and improve matching reliability. Building upon this, the memory-guided trajectory cues within a bounded-size sliding window are further formulated as adaptive motion factors and jointly optimized with detection measurements and geometric point-cloud constraints in a unified factor graph to refine tracked trajectories. Extensive experiments on the KITTI and nuScenes benchmarks demonstrate significant improvements in both MOT performance and object localization accuracy compared with state-of-the-art methods. The proposed framework achieves the highest HOTA of (85.97%, 81.22%) and MOTP of (92.05%, 87.28%), with the lowest IDSW on the KITTI validation/test sets, respectively, while reducing translational trajectory errors by up to 75.15% compared with existing methods. For the nuScenes benchmark, the proposed framework further obtains the highest MOTA of 75.94%, the lowest AMOTP of 34.61%, and the lowest fragmentation number of 128, validating its robustness and generalization capability in complex urban environments.