Opportunistic Dynamic Architecture for Class-Incremental Learning

Continual learning has attracted increasing attention over the last few years, as it enables to continually learn new tasks over time, which has significant implication to many real-world applications. A large number of continual learning techniques are proposed and achieve promising performance; however, many of them commit to a fixed, large architecture at the beginning, which can waste the memory space and incur high training cost. To directly tackle this challenge, we propose an Opportunistic Dynamic Architecture, ODA, based on mixture of experts. ODA can automatically grow with more experts for new incoming tasks and opportunistically shrink by merging experts with similar weights. We evaluated ODA on three commonly used datasets: CIFAR-100, CUB, and iNaturalist, and compared against eight existing continual learning techniques. ODA not only outperforms these techniques but does so with a parameter size that is slightly smaller on average, maintaining memory efficiency without compromising accuracy. Furthermore, ODA achieves this with only around 16% of the training time across all datasets when updating for each new task, making it a highly resource-efficient solution for continual learning applications.