A holistic multimodal approach for real-time anomaly detection and classification in large-scale photovoltaic plants

• This paper proposes a holistic approach for photovoltaic plant monitoring, combining data acquisition, segmentation, geolocation, anomaly detection, real-time classification, and post-inspection into a unified production workflow. • Multimodal data fusion (RGB and IR) is employed to enhance anomaly detection accuracy, leveraging the strengths of both modalities. • The improved ConvNext-Tiny architecture, optimized via Bayesian search, achieved 99% precision, recall, and F1-score for binary classification, and 85% for multi-class classification, proving its robustness for real-time PV plant monitoring. • Solar module segmentation, based on YOLOv7, achieved a [email protected] of 94% for RGB data and 99% for IR data, ensuring high accuracy across modalities. • Real-time inference is achieved with 14 ms for segmentation and 24 ms per image for classification, ensuring efficient and responsive anomaly detection. • The approach introduces a production-ready pipeline optimized for real-time execution on edge devices, enabling seamless integration into operational PV plants with low-latency inference. This paper presents a holistic multimodal approach for real-time anomaly detection and classification in large-scale photovoltaic plants. The approach encompasses segmentation, geolocation, and classification of individual photovoltaic modules. A fine-tuned Yolov7 model was trained for the individual module’s segmentation of both modalities; RGB and IR images. The localization of individual solar panels relies on photogrammetric measurements to facilitate maintenance operations. The localization process also links extracted images of the same panel using their geographical coordinates and preprocesses them for the multimodal model input. The study also focuses on optimizing pre-trained models using Bayesian search to improve and fine-tune them with our dataset. The dataset was collected from different systems and technologies within our research platform. It has been curated into 1841 images and classified into five anomaly classes. Grad-CAM, an explainable AI tool, is utilized to compare the use of multimodality to a single modality. Finally, for real-time optimization, the ONNX format was used to optimize the model further for deployment in real-time. The improved ConvNext-Tiny model performed well in both modalities, with 99% precision, recall, and F1-score for binary classification and 85% for multi-class classification. In terms of latency, the segmentation models have an inference time of 14 ms and 12 ms for RGB and IR images and 24 ms for detection and classification. The proposed holistic approach includes a built-in feedback loop to ensure the model’s robustness against domain shifts in the production environment.