Adaptive Segmentation of Thermal Anomalies in Buildings Using Human-in-the-Loop Active Learning Capsule Networks

Buildings account for 40% of energy consumption in the U.S., with significant energy losses stemming from poor insulation and leaks. To meet decarbonization goals by 2050, efficient methods for diagnosing and retrofitting thermal anomalies are essential. Infrared thermography (IRT), combined with emerging technologies like computer vision and deep learning, offers the potential for automating thermal anomaly detection and segmentation. However, challenges such as building diversity, scenario variations, and labor-intensive image annotation hinder model reliability and robustness. This study proposes a human-in-the-loop active learning approach to fine-tune the pretrained Capsule-based network (CapsLab) to enhance adaptability to new building scenarios efficiently. Using a Query-by-Committee (QBC) strategy, the method selects the most informative thermal images from a target dataset for expert verification and iterative model refinement. To streamline the expert annotation verification process, a weakly annotation strategy is introduced supporting human-in-the-loop training by Simple Linear Iterative Clustering (SLIC)-based superpixel segmentation and scribble prompts for efficient labeling. The pretrained CapsLab model is iteratively fine-tuned using these verified annotations to enhance segmentation performance. Three fine-tuning methods, including parameter freezing strategies, are evaluated for optimal results. This iterative workflow reduces annotation effort, improves model adaptability, and improves the precision of thermal anomaly segmentation, facilitating energy-efficient building retrofitting and maintenance.