Prompt Driven Multimodal Large Language Models for Concrete Defect Identification

Detecting surface defects in concrete structures is essential for ensuring structural safety; however, traditional methods have limitations, including subjectivity and heavy data dependency. This study proposes using pretrained multimodal Large/Small Language Models (LLM/SLM), specifically GPT-4o and Gemini 2.5-Flash, for concrete defect detection without additional fine-tuning. Structured prompts (S-Prompt) were employed to direct the task instructions and to elicit structured JSON outputs. The models’ performances were evaluated in both zero-shot and few-shot scenarios, the latter including visual examples. Performance was benchmarked against a fine-tuned YOLOv8 model, employing standard detection metrics alongside newly introduced metrics: Class Presence Accuracy (CPA), Relaxed Localization Recall (RLR), and Unmatched Prediction Rate (UPR). Experimental results demonstrated that even in a zero-shot setting, the LLM/SLM models meaningfully identified defect presence, type, and approximate location (at an IoU threshold of 0.1) (GPT-4o: [email protected]=0.487, CPA-F1=0.667). The few-shot scenario improved certain metrics ([email protected], CPA-F1, [email protected], etc.) but exhibited trade-offs including increased UPR and reduced inference speed. Although the proposed method showed lower performance than YOLOv8 in precise localization tasks (higher IoU thresholds), it demonstrated significant potential as a training-free pipeline suitable for initial screening purposes without the burden of extensive data labeling and retraining.