Large Vision Language Models for Anomaly Detection in Aircraft Inspection with Inspectors in the Loop

Visual inspection is mainly performed by human inspectors in aviation. However, the performance of inspectors in manual inspections is inconsistent due to human factors. Therefore, several Artificial intelligence (AI) methodologies have been proposed to improve human inspectors’ performance. While these methodologies are promising, a few usability and development hurdles still need to be addressed. Therefore, the researchers propose a novel LVLM-AD-IL framework: Large Vision Language Models for Anomaly Detection in aircraft inspection with Inspectors in the Loop. This framework leverages LVLMs’ multimodal generic knowledge and integrates the inspectors’ expert knowledge for anomaly inspection tasks. This framework was developed in ChainForge, which is an open-source visual programming tool designed to facilitate the assessment of LLMs’ prediction performance. The LVLM-AD-IL framework does not require training and can perform logical and structural analysis on multi class anomaly detections. Using this framework, a case study was conducted with two classes: fuel tank and flap attachment. This research measured the accuracy and consistency of prediction in anomaly determination and anomaly reasoning, examining these aspects through ten iterations. The results found that the GPT 4.0 model performed well in class 1, both in anomaly determination and classification, as well as in reasoning. In contrast, the GPT 3.5 model outperformed GPT 4.0 in class 2 in anomaly determination classification, but the GPT 4.0 model performed relatively better in anomaly reasoning.