α-ADARTS: Alpha channel attention for differentiable architecture search

The use of gradient guided search in Differentiable neural architecture search (DARTS) significantly saves the expenses of neural network architecture search. However, DARTS suffers from poor architecture stability due to the occurrence of substantial weight-free operations in later stages of search. A novel long scope attention mechanism was proposed to improve differentiable neural architecture search, termed α-ADARTS. The attention mechanism assigns weights to channels, selectively incorporating important ones for efficient and stable search. Different from the existing methods limited to feature maps, its scope includes the feature maps and the architecture of computing cells. In addition, we explored the relationship between the depth of computing units and the proportion of partial channel connections. By utilizing this, the stability of long-term network search has been further enhanced. Experimental results demonstrate that test error on CIFAR10 and CIFAR100 were only 16.07% and 2.49% cost only 0.06 GPUdays. Long term search experiments have shown that compared to other methods, the proposed method can effectively inhibition weight-free operations and improve stability. Code accessible at https://github.com/Cxllxc/alpha-ADARTS.