CMINet: A Unified Cross-Modal Integration Framework for Crop Classification From Satellite Image Time Series

Accurate automatic interpretation of crops is crucial for agricultural monitoring and food security assessment. In recent years, there has been a rise in platforms generating multimodal and multitemporal remote sensing images at an unprecedented speed. These images provide rich temporal, spatial, and spectral information, enabling more comprehensive land cover classification. Therefore, single-modal crop classification can benefit from complementary modalities. However, given the distinct characteristics of different modality sensors, enhancing the performance of deep networks by integrating diverse modality data remains a significant challenge. Unlike previous methods, this article proposes a two-stage fusion network named CMINet for crop classification from satellite image time series (SITS). Specifically, we adopt a decoupled framework to encode the spatial and temporal features of SITS. At each level of the spatial encoder, we design a cross-modal transfer module (CMTM) to complement bimodality branch features by transferring knowledge from one modality to rectify the features of another modality. After rectified feature pairs are encoded by a temporal encoder, we develop a cross-attention fusion module (CAFM) to conduct adequate context exchange before merging. The seamless combination of these novel designs forms a robust multimodal representation, outperforming the state-of-the-art methods on two public multimodal crop classification datasets. Compared to existing methods, our CMINet improves at least 1.1% OA, 1.7% mF1, and 2.0% mIoU on the PASITS-R dataset and 2.6% OA, 1.1% mF1, and 2.8% mIoU on the South Sudan dataset.