Few-Shot Defect Recognition for New Energy Equipment via Multimodal Harmony

Internet of Things (IoT) technologies have been applied to fault detection in new energy equipment, which is crucial for ensuring the stable operation of energy systems. However, existing approaches typically rely on large amounts of labeled data to train intelligent algorithms, which are difficult to obtain in real-world. To address this challenge, this paper proposes a novel multi-modal few-shot defect recognition framework for new energy equipment, enabling data-efficient defect recognition in real-world scenarios through multi-modal harmony. Unlike previous multi-modal few-shot methods, it eliminates the necessity of pairwise similarity calculations, thereby simplifying the training and inference processes. Our approach trains a shared classifier through text and visual modality features integration. Initially, we extract feature vectors using text and visual encoders and then map them to common feature space. Subsequently, the vectors of these two modalities are used to train a shared classifier, which aids in the simultaneous learning of corresponding visual representations and conceptual information. Furthermore, to enhance interaction between modalities, the feature vectors of the text prompts are used to initialize the classifier weights, thereby promoting cross-modal consistency. Experiments on datasets of wind turbine blades and solar cells show that combining the two modalities improves recognition accuracy by up to 19.13% and 28.52% compared to other multi-modal few-shot methods. Despite its reliance on prompt quality, the approach provides an effective and scalable solution for defect recognition.