Defect Classification and Localization in Material Extrusion with Multi-Modal Large Language Models

This study explores applying multi-modal large language models (MLLMs) for defect classification and localization in material extrusion (MEX) additive manufacturing (AM), focusing on common defects such as stringing and layer misalignment. Unlike traditional machine learning methods that require extensive labelled datasets, MLLMs offer the ability to combine textual and visual data to streamline defect detection. This study evaluates the performance of two state-of-the-art MLLMs, GPT-4o and Gemini-Pro-Vision, across varying sampling temperatures and prompt engineering techniques. The experiments demonstrated that GPT-4o achieved a classification accuracy of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{9 1. 4 2 \%}$</tex> with refined prompts and low sampling temperature settings, significantly outperforming Gemini-ProVision. However, both models struggled with defect localization, with a maximum accuracy of only <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$16.27 \%$</tex>. These findings suggest that while MLLMs show potential in automated defect detection, there is a critical need for spatial reasoning and localization accuracy improvements. Future work should focus on enhancing model training and integrating advanced image processing techniques to improve the applicability of MLLMs in real-time quality control for MEX.