Physics-Informed Deep Learning-Based Monocular Vision and IMU Fusion for Extracting Position and Heading Data of Multirotors

Multirotor Unmanned Aerial Vehicles (UAV)s flight performance rely on accurate estimation of their attitude and position for stable and robust control, which is essential for precise path following and management. Traditional methods for estimating six Degrees of Freedom (DoF) parameters, such as position, orientation, and velocity, often combine data from Global Navigation Satellite System (GNSS), Inertial Measurement Unit (IMU), and vision sensors using complex algorithms for fusion and filtering. This approach can lead to challenges in real-time processing and robustness to environmental variations. In this paper, we propose a novel approach that leverages deep learning and incorporates a physics-informed loss function based on monocular vision data to enhance the accuracy of these sensors by eliciting position and heading data from a trained model of flow detection, enhanced with a mathematical model of optical flow. We implement a fusion of camera, IMU, magnetometer, and ultrasonic sensors for robust and accurate 6 DoF data acquisition in multirotor systems. Our proposed framework utilizes Convolutional Neural Networks (CNN)s to directly learn the mapping between sensor data and 6 DoF parameters. By jointly processing visual and inertial data in an end-to-end manner, our model exploits the complementary information provided by each sensor modality, thereby enhancing estimation accuracy and robustness to external disturbances. We demonstrate the effectiveness of our approach through extensive experiments on real-world multirotor platforms, show-casing significant improvements in pose and motion estimation accuracy compared to conventional methods. Our fusion deep learning framework offers a promising pathway for enhancing the autonomy and performance of multi rotor UAVs.