Robust and Efficient Multi-Fidelity Neural Architecture Search with Zero-Cost Proxy-Guided Local Search

Training-free metrics, also known as zero-cost (ZC) proxies, enable efficient exploration in Neural Architecture Search (NAS) but are less effective than training-based metrics like validation accuracy in identifying high-performance networks. In this article, we investigate the effectiveness of ZC proxies by taking a deeper look into their fitness landscapes utilizing Local Optima Networks. We introduce MF-NAS, a two-stage NAS framework that first employs a ZC proxy-guided local search algorithm to explore the search space. Networks with the highest ZC scores are then fed into the Successive Halving (SH) algorithm to identify the top-performing architecture. However, we observe considerable performance gaps when different ZC metrics are employed. Analyzing those MF-NAS variants with Search Trajectory Networks, we find that high-performance networks could be encountered early or midway through the ZC proxy-guided local search process, making selection based solely on the highest ZC scores ineffective in certain cases. To address this issue, we propose the R-MF-NAS framework, where the selected networks for SH include not only those with the highest ZC scores but also promising solutions encountered during the local search stage. Experiments on diverse NAS benchmarks demonstrate the superiority of both MF-NAS and R-MF-NAS over state-of-the-art methods under a strict budget.