Multimodal Satellite-Field Data Integration for Scalable Crop-Type Mapping With Limited Labels

Crop type mapping is crucial for improving crop yield predictions, informing agricultural policy decisions, and addressing the impacts of climate change. Despite achieving state-of-the-art performance, current transformer-based architectures that use Satellite Image Time Series (SITS) require costly labelled data and struggle to generalize to new regions. To address this limitation, this paper presents a method that leverages multimodal information from field images and satellite data to address the scarcity of reference data in crop type mapping. The proposed approach consists of three main steps: (1) multimodal data preparation, (2) multimodal <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">pseudo</i>-labelling, and (3) SITS-based crop type mapping. The first step processes both data sources for facilitating their integration, while the novelty lies in the second step, where additional <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">pseudo</i>-labels are generated by classifying crop parcels with both SITS and field image availability, leveraging multimodal information to produce reliable label predictions. Finally, the third step produces a crop type map for the entire study area using the SITS and the enriched training set. Experimental results at the European level, using the 2018 Land use and land cover survey (LUCAS) field images paired with Sentinel-2 SITS, demonstrate the effectiveness of integrating satellite top-view data with ground-level field images. The <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">pseudo</i> labelled data achieves performance comparable to models trained on the complete labelled dataset, while substantially reducing the need for extensive labelled data collection.