Spatio-temporal-text fusion for hierarchical multi-label crop classification based on time-series remote sensing imagery

Recent advances in deep learning have enhanced crop classification, yet current research still underutilizes the hierarchical information of crop types, limiting classification accuracy. As categories are subdivided, the sample imbalance intensifies, posing a challenge to fine classification of crops. To address this, we propose the Class Semantic Guided Hierarchical Segmentation Framework (SemHi framework) for satellite image time series (SITS) crop classification. This framework effectively leverages the hierarchical information in the class system and outputs classification results at each level in an end-to-end manner. The SemHi framework comprises four modules: (1) The backbone that learns the spatio-temporal features; (2) The label embedding module, a core component that guides the feature learning through the fusion of hierarchical structure and textual representations; (3) The prototype distance measurement module to enhance class separation and reduce within-class variation; (4) The hierarchical logic regularization module, which enables multi-granularity crop predictions and strengthens the hierarchical logic between them. The validation results on the public crop classification dataset show that the SemHi framework with multi-dimensional fusion of spatio-temporal-text information significantly improves the performance of all state-of-the-art (SOTA) networks on fine-grained classes, with an overall accuracy improvement of 0.48% to 13.86%. Furthermore, experiments on a remote sensing classification dataset demonstrate the framework’s generality and potential for broader applications in remote sensing tasks.