Modality Distillation Using a SAM-Guided Multimodal Teacher for Unimodal Wildfire Segmentation and Temperature Prediction

Wildfires are one of the world’s most devastating natural disasters that affect the environment, communities, and more critically, humans that live in and around those communities. Due to the threat of large-scale destruction in landscapes and human inhabited areas, it has become increasingly more important to develop wildfire detection, management, and suppression strategies to mitigate and prevent these negative outcomes. Wildfire research encompasses many different areas. Most notably, the development of communication, navigation, remote sensing, and monitoring systems. In wildfire monitoring, limitations discovered in-ground and satellite observation have shifted the focus toward Unmanned Aerial Vehicle (UAV) based wildfire research, which has proven promising. Modern UAV-based wildfire detection methods primarily focus on using artificially intelligent deep neural networks with collected aerial wildfire imagery to classify, localize, or segment wildfires. While classification tasks are capable of achieving high accuracy in many domains with single modality input, such as RGB imagery only, segmentation tasks in domains containing dynamically changing environments can become more complicated, requiring multiple input types to prove valid in real-world applications. Additionally, wildfire research in computer vision largely focuses on the classification only of fire within an image or binary segmentation of fire itself. These tasks have become more trivial in recent years, and research needs to move toward more detailed understanding of wildfire using computer vision approaches. The practical application of wildfire segmentation is that it isolates fire location within an image, which can be used to identify different intensity regions or hotspots of the fire. With this knowledge, dangerous regions of the fire are found and this information can be communicated to protect firefighters or other personnel from getting too close to dangerous areas of the natural disaster. This paper explores binary wildfire segmentation and pixel-wise temperature prediction, in degrees Celsius, using radiometric temperature-labeled ground truth. A semi-supervised modality distillation approach is used, where a SAM-guided multimodal segmentation network is trained alongside a unimodal network, transferring knowledge for both segmentation and temperature regression. With this work, wildfire research can progress to more complex raw temperature inference, using only RGB image data input, potentially eliminating the use of IR sensors in the future.