E2E: Onboard satellite real-time classification of thermal hotspots events on optical raw data

Abstract Nowadays, the use of Machine Learning (ML) onboard Earth Observation (EO) satellites has been investigated for a plethora of applications relying on multispectral and hyperspectral imaging. Traditionally, these studies have heavily relied on high-end data products, subjected to extensive pre-processing chains natively designed to be executed on the ground. However, replicating such algorithms onboard EO satellites poses significant challenges due to their computational intensity and need for additional metadata, which are typically unavailable on board. Because of that, current missions exploring onboard ML models implement simplified but still complex processing chains that imitate their on-ground counterparts. Despite these advancements, the potential of ML models to process raw satellite data directly remains largely unexplored. To fill this gap, this paper investigates the feasibility of applying ML models directly to Sentinel-2 raw data to perform thermal hotspot classification. This approach significantly limits the processing steps to simple and lightweight algorithms to achieve real-time processing of data with low power consumption. To this aim, we present an end-to-end (E2E) pipeline to create a binary classification map of Sentinel-2 raw granules, where each point suggests the absence/presence of a thermal anomaly in a square area of 2.5 km. To this aim, lightweight coarse spatial registration is applied to register three different bands, and an EfficientNet-lite0 model is used to perform the classification of the various bands. The trained models achieve an average Matthew’s correlation coefficient (MCC) score of 0.854 (on 5 seeds) and a maximum MCC of 0.90 on a geographically tripartite dataset of cropped images from the THRawS dataset. The proposed E2E pipeline is capable of processing a Sentinel-2 granule in 1.8 s and within 6.4 W peak power on a combination of Raspberry PI 4 and CogniSat-XE2 board, demonstrating real-time performance.