Category‐instance distillation based on visual‐language models for rehearsal‐free class incremental learning

Abstract Recently, visual‐language models (VLMs) have displayed potent capabilities in the field of computer vision. Their emerging trend as the backbone of visual tasks necessitates studying class incremental learning (CIL) issues within the VLM architecture. However, the pre‐training data for many VLMs is proprietary, and during the incremental phase, old task data may also raise privacy issues. Moreover, replay‐based methods can introduce new problems like class imbalance, the selection of data for replay and a trade‐off between replay cost and performance. Therefore, the authors choose the more challenging rehearsal‐free settings. In this paper, the authors study class‐incremental tasks based on the large pre‐trained vision‐language models like CLIP model. Initially, at the category level, the authors combine traditional optimisation and distillation techniques, utilising both pre‐trained models and models trained in previous incremental stages to jointly guide the training of the new model. This paradigm effectively balances the stability and plasticity of the new model, mitigating the issue of catastrophic forgetting. Moreover, utilising the VLM infrastructure, the authors redefine the relationship between instances. This allows us to glean fine‐grained instance relational information from the a priori knowledge provided during pre‐training. The authors supplement this approach with an entropy‐balancing method that allows the model to adaptively distribute optimisation weights across training samples. The authors’ experimental results validate that their method, within the framework of VLMs, outperforms traditional CIL methods.