Event-RGB fusion for spacecraft pose estimation under harsh lighting

• Developed a sensor rig to capture optically and temporally aligned event-RGB data to evaluate spacecraft pose estimation algorithms. • Released the first public temporally and optically aligned event-RGB dataset for evaluating spacecraft pose estimation methods with both synthetic and real sequences. The real sequences are captured under harsh lighting effects such as: glare, blooming and lens flares. • Proposed a learning-free test-time event-RGB fusion method which exploits the precise alignment achieved by the aligned sensor rig and fuses 2D-3D correspondences across input sources via the classical RANSAC algorithm to tackle the harsh lighting challenge. • The proposed method is trained only on synthetic data and performs learning-free test-time fusion incorporating uncertainty measures to estimate more accurate poses than single channel methods. Spacecraft pose estimation is crucial for autonomous in-space operations, such as rendezvous, docking and on-orbit servicing. Vision-based pose estimation methods, which typically employ RGB imaging sensors, is a compelling solution for spacecraft pose estimation, but are challenged by harsh lighting conditions, which produce imaging artifacts such as glare, over-exposure, blooming and lens flare. Due to their much higher dynamic range, neuromorphic or event sensors are more resilient to extreme lighting conditions. However, event sensors generally have lower spatial resolution and suffer from reduced signal-to-noise ratio during periods of low relative motion. This work addresses these individual sensor limitations by introducing a sensor fusion approach combining RGB and event sensors. A beam-splitter prism was employed to achieve precise optical and temporal alignment. Then, a RANSAC-based technique was developed to fuse the information from the RGB and event channels to achieve pose estimation that leveraged the strengths of the two modalities. The pipeline was complemented by dropout uncertainty estimation to detect extreme conditions that affect either channel. To benchmark the performance of the proposed event-RGB fusion method, we collected a comprehensive real dataset of RGB and event data for satellite pose estimation in a laboratory setting under a variety of challenging illumination conditions. Encouraging results on the dataset demonstrate the efficacy of our event-RGB fusion approach and further supports the usage of event sensors for spacecraft pose estimation. To support community research on this topic, our dataset has been released publicly.