Semi-Dynamic Ensemble for Evolutionary NAS in Continual Learning

Performance improvements of Deep Learning techniques have recently grown interest on new AI applications in increasingly restricted scenarios. On one side, with larger neural networks demanding even longer training as tasks become more complex, recent models have studied techniques to learn without forgetting, avoiding retraining from scratch. On the other side, energy and memory constraints on scenarios like edge computing have raised concern about efficient models, adapting their own architecture to retain learning capacity while optimizing the usage of multiple resources. Addressing those demands, we propose the first multi-objective evolutionary Neural Architecture Search (NAS) strategy to excel on continual learning scenario, SEENAS, while being flexible to memory-restricted applications. Therefore, we adopt an under-explored NAS strategy on Continual Learning, adopting a multi-objective genetic algorithm to learn the best architecture given an incremental task and a complexity-related criterion. Similarly, we define a semi-dynamic deep-ensemble-based search space, of limited size, to improve the classification robustness, proposing a novel voting strategy and ensemble management method. We evaluate the proposed strategies on the CIFAR100 and ImageNet-100 datasets under limited computational power, reporting competitive performance among NAS solutions for continual learning.