Video stabilization based on deep learning and manifold optimization

As a crucial preprocessing step for mobile vision applications, video stabilization technology often suffers from issues such as interference from dynamic objects and motion estimation distortions caused by the coupling of complex backgrounds in scenarios like drone aerial photography and handheld shooting. Current mainstream methods exhibit two key limitations when dealing with unstructured motion scenes: motion parameter estimation is sensitive to feature sparsity, leading to error accumulation; and local filtering-based smoothing struggles to balance temporal consistency and spatial smoothness of motion trajectories, often resulting in motion artifacts in dynamic and complex scenes. To address these issues, inspired by self-supervised motion estimation based on test-time training, we propose a novel video stabilization approach based on a two-stage architecture. In the first stage, we estimate inter-frame 3D camera motion based on the idea of test time training combined with optical flow. The second stage serves as the correction phase, where the estimated sequence of rotation matrices is globally optimized using Riemannian geometry on the manifold, while the translation sequence is smoothed with a Gaussian filter. To enhance the correlation between motion estimation and smoothing modeling, we introduce a dual-manifold constraint on camera poses during the motion estimation stage, incorporating both consistency and smoothness constraints. Extensive experiments conducted across various video scenarios demonstrate that our method significantly improves video stabilization performance compared to baseline approaches. Notably, in complex environments such as low-texture indoor scenes, our approach achieves approximately a 6% improvement in stabilization performance over the baseline model.