Hardware Latency-Aware Differential Architecture Search: Search for Latency-Friendly Architectures on Different Hardware

As a result of its low search cost, Differentiable Architecture Search (DARTS) has recently received a lot of interest. Nowadays, most methods based on DARTS only focus on improving a single indicator (e.g., accuracy), making the search process more inclined to complex networks with more robust representational capabilities. Hence, the architectures searched by these methods tend to own high latency, leading to DARTS in some low-latency scenarios or edges with limited computing power, and on-device deployment becomes difficult. To deal with this challenge, we propose a Hardware Latency-Aware Differentiable Search (HL-DARTS) algorithm. This algorithm designs a multi-layer regression network that uses the soft attention mechanism to predict the latency on the corresponding hardware devices, thus adding a differentiable latency loss term based on the DARTS algorithm. We further propose an adaptive constraint amplitude—a mechanism for balancing accuracy and latency while searching for a latency-friendly architecture for a given hardware device. We conduct ablation experiments on different datasets and different hardware devices. The experimental results show that HL-DARTS can find the ideal architecture for different hardware devices and that this architecture is also broadly applicable to various datasets.