Autonomous Identification and Localization of Battle Damage on Aircraft Using Infrared-Based Sensors

Autonomous in-flight inspections of aircraft battle damage by unmanned aircraft systems (UAS) can provide critical information to operators while improving safety and eliminating the need for visual in-flight inspections with other manned aircraft. This work centers on the use of an UAS with commercial off-the-shelf (COTS) infrared (IR)-based sensors to autonomously identify battle damage on aircraft. The proposed solution employs a convolutional neural network (CNN), Simultaneous Localization and Mapping (SLAM), and a pixel-to-world point converter to detect, classify, and localize aircraft damage in real-time and onboard. This paper presents an experiment that autonomously detected and identified aircraft battle damage on a T -37 airplane wing in day and night conditions with a 58.02% success rate on average. Accurate localization, though verified experimentally with different sensors, was not fully tested due to mediocre estimations of sensor intrinsics. This methodology has promise in many applications related to battle awareness due to its improvements over standard red, green, blue, depth (RGB-D) sensor packages under poor visibility conditions. Potential future applications of this work include remotely piloted aircraft (RPA) using existing sensor packages for aircraft monitoring and fighter aircraft wingmen passively inspecting one another. This technology could also be used by commercial aircraft and spacecraft to quickly and confidently identify structural damages or other anomalies. Future work includes increasing the quality and quantity of training for the CNN, further characterizing the sensors, adapting the system to inspect moving aircraft, and fusing the IR sensors with standard color cameras.