DNAS: Depth-First Neural Architecture Search

The key challenge of neural architecture search (NAS) methods lies in efficiently exploring search spaces. To solve this problem, Breadth-First Search (BFS) method uses two trees to represent a search space, and performs bi-level BFS on these trees to find the optimal network architecture. However, the BFS method did not discover the best network in NAS-Bench-201, and its search efficiency is still not high enough. In this work, we propose a one-shot method called Depth-first Neural Architecture Search (DNAS) to efficiently explore the best architecture. Given a search space with <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$N$</tex>-candidate operations, we represent it as a single <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$N$</tex>-ary tree and employ depth-first search on this tree to explore high-performance network architectures while significantly reducing the resource consumption during the exploration. While the BFS method explores multiple networks simultaneously in each exploration, DNAS efficiently explores along only one direction at a time, eliminating the need to synchronize the search processes of multiple networks. The proposed DNAS, performed on a single RTX 2080Ti GPU, finds the optimal architectures for CIFAR-10 and ImageNet16-120 in 0.3 hour on NAS-Bench-201, and in 1.0 hour on CIFAR-100. While finding the best network, the search time of the DNAS method has been reduced by around 2 to 8 GPU hours as compared to other well-performed methods, especially reducing the search time by 91.67 % when compared to the BFS method. The best searched network was further transferred to medical image classification tasks and achieved high classification accuracy across multiple datasets. In addition, the results of ablation experiments substantiate the effectiveness and efficiency of the proposed method. Our source code is available at: https://github.com/Bob5090/DNAS.