Zero-Shot Anomaly Detection in Laser Powder Bed Fusion Using Multimodal Retrieval-Augmented Generation and Large Language Models

Abstract Additive manufacturing (AM) enables the fabrication of complex designs while minimizing waste, but faces challenges related to defects and process anomalies. This study presents a novel multimodal retrieval-augmented generation (RAG)-based framework that automates anomaly detection across various additive manufacturing processes leveraging retrieved information from the literature, including images and descriptive text, rather than training datasets. This framework integrates text and image retrieval from the scientific literature and multimodal generation models to perform zero-shot anomaly identification, classification, and explanation generation in a laser powder bed fusion (L-PBF) setting. The proposed framework is evaluated on four L-PBF manufacturing datasets from the Oak Ridge National Laboratory, featuring various printer makes, models, and materials. This evaluation demonstrates the framework's adaptability and generalizability across diverse images without requiring additional training. Comparative analysis using Qwen2-VL-2B and GPT-4o-mini as multimodal large language model (MLLM) within the proposed framework highlights that GPT-4o-mini outperforms Qwen2-VL-2B and proportional random baseline in manufacturing anomalies classification. Additionally, the evaluation of the RAG system confirms that incorporating retrieval mechanisms improves average accuracy by 12% by reducing the risk of hallucination and providing additional information. The proposed framework can be continuously updated by integrating emerging research, allowing seamless adaptation to the evolving landscape of AM technologies. This scalable, automated, and zero-shot-capable framework streamlines AM anomaly analysis, enhancing efficiency and accuracy.