Dynamic inference for on-orbit scene classification with the scale boosting model

Existing scene classification methods allocate the same computational resources, i.e., all model parameters in the neural network, to each remote sensing image whenever from any geographic scene. However, this might be redundant for images of certain scenes that are easy to discriminate, e.g., homogeneous scenes. This observation motivates us to propose an efficient method for on-orbit scene classification, namely, the Scale Boosting Model (SBM). Specifically, during the training process, the SBM is built as a set of different scale learners in a scale-increasing manner, each of which is used to learn and classify image features at a specific scale. During inference, the scale learners in the SBM will be selectively run in a scale-increasing manner and automatically decide when to exit early or expand the computation according to the scene complexity. In addition, by replacing the backbone of the scale learner, the SBM could provide a deployment possibility for computationally limited models for on-orbit processing, thereby reducing their computational requirements. Extensive experiments on UC Merced Land Use, NWPU-RESISC45 and RSD46-WHU datasets show that the SBM achieved a more effective classification performance more efficiently. • A novel model for learning multi-scale representations of scenes is proposed. • SBM dynamically allocates FLOPs according to the complexity of scenes. • The difficulty of classification of different scenes is quantitatively revealed. • SBM provides an efficient framework for the on-orbit limited models.