Mixture-of-Experts as Continual Knowledge Adapter for Mobile Vision Understanding

Continual machine learning in the context of limited computational resources and data availability is critical in the connected digital world. Current intelligent applications predominantly rely on deep learning models requiring labor/computation-intensive training. These models often struggle to adapt effectively to new data while preserving performance on previously learned knowledge. In this paper, we introduce a lightweight method for continual knowledge adaptation that can address these challenges. To prevent disruption of the existing services, we propose a Mixture-of-Experts (MoE) adapter that integrates seamlessly with the existing vision model to encode new data. The weights of the original model are kept fixed during the adaptation process, ensuring the preservation of previously learned knowledge. The MoE technique enables scaling up the parameters of the adapter while maintaining a relatively low computation, making it fit for constrained devices in mobile computation scenarios. Furthermore, to enhance learning efficiency and accelerate convergence with new data, we implement a knowledge fusion mechanism that facilitates interaction between the existing knowledge and the information extracted from new data. The timing of employing the fusion module is further investigated. We find that it is conducive in scenarios where the task's performance requirements are enhanced. The MoE adapter and knowledge fusion module are integrated at each stage with minimal trainable parameters, efficiently optimizing resource usage. Extensive experiments and ablation studies validate the effectiveness of the proposed method. Specifically, the proposed method prevents an accuracy drop of 43.02% on the previous data compared to the continual train method, while achieving an accuracy of 44.81% on the new data, which is even 0.34% higher than fully training a new model.