A Surrogate Model With Multiple Comparisons and Semi-Online Learning for Evolutionary Neural Architecture Search

Evolutionary Neural Architecture Search (ENAS) refers to techniques to automatically design neural networks through an evolutionary framework. Although ENAS has great potential, its application in practice can be very computationally expensive, especially for the design of Deep Neural Networks (DNNs). The use of surrogate models (computationally cheap models that approximate computationally expensive objective functions) is a viable option to reduce the design cost. Research on surrogate models for ENAS has recently shifted from directly predicting the performance of architectures to predicting the outcome of pairwise comparisons. The present paper proposes a novel surrogate-assisted ENAS algorithm, namely a surrogate model with multiple comparisons and semi-online learning (SMCSO). The proposed SMCSO tests the performance of a newly generated architecture against that of a pool of promising architectures (multiple comparisons). This way, the algorithm exhibits robust behaviour, which alleviates the risk of performing wrong selections due to the uncertainty introduced by the surrogate. This logic is then naturally embedded within the particle swarm optimisation (PSO) logic, where newly generated architectures would need to be compared against the local best particle (in our case, there is a pool of local best particles). Furthermore, the initially trained surrogate model is run within the PSO and then retrained every time a prearranged number of local best replacements occur. We call this training technique semi-online learning. It allows us to directly control the trade-off between accuracy and the computational overhead of the surrogate model. Experimental results show that on the NAS-Bench-101 and the NAS-Bench-201, the architectures detected by the proposed SMCSO achieved 94.15% and 94.33% accuracy, respectively, thus, establishing a new state-of-the-art surrogate-assisted ENAS.