Comparative study of Neural Architecture Search methods: Random Search, RNN + Reinforcement Learning, and Evolutionary Algorithms on CIFAR-10

In the past few years, Neural Architecture Search (NAS) is an attractive technique that promised automatic design for high-performance neural networks. However, NAS methods considerably vary in search efficiency, as well as model performance. In this paper, we have compared three mainstream NAS methodologies: Random Search, RNN + RL (Recurrent Neural Networks and Reinforcement Learning based method), and the Evolutionary Algorithm suggested in this paper. The experiments were on the CIFAR-10 dataset and each method’s accuracy, training time, and parameter count were analyzed. Research methodology further involves experimental analysis of these three NAS methods over the CIFAR-10 dataset, with particular attention proposed to enhance the performance of the Evolutionary Algorithm through improving the encoding and mutation strategies. The Evolutionary Algorithm has good searching efficiency and is hence deemed ‘environmentally friendly’ RNN + RL is significantly better model-and parameter-wise. Implementing a dynamic mutation rate strategy with the Evolutionary Algorithm enhances the searching efficiency and model performance of the Evolutionary Algorithm. Successive to mark the evolution of fine-tuning of the Evolutionary Algorithm with other NAS techniques will further enhance it, finally unleashing its full potential in advancing automated design techniques for deep learning models.