Boosting explainability of visual inspection via multihead vision-language model

We proposed an explainable general visual inspection model using Vision-Language Model (VLM) with few-shot images of non-defective or defective products, along with explanatory texts that serve as decision criteria. Existing VLMs are not trained on specific tasks such as visual inspection; thus, we fine-tuned VLM on our own dataset for visual inspection. Also, for new products, existing method employed In-Context Learning, which allows the model to perform inspections with an example of non-defective or defective products and the corresponding explanatory texts with visual prompts. However, existing VLMs are trained on CrossEntropyLoss, which cannot consider the distance of numeric values, this is the same during the fine-tuning. Thus, we modified the existing single-head VLM into a multi-head VLM to address visual inspection and anomaly detection simultaneously. Task-specific heads can be trained on task-specific losses (e.g., for anomaly detection, L1 loss and GIoU Loss are used). Experimental results showed that our approach did not improve the pixel-level AUROC, but our approach can generate the output text, expressing the inspection criteria of the model. Also, detailed analysis showed that it is too difficult to train additional task-specific head on the limited training dataset. Our research provides novel insights for training the task-specific heads.