Zero-Cost Operation Scoring in Differentiable Architecture Search

Abstract

Differentiable neural architecture search (NAS) has attracted significant attention in recent years due to its ability to quickly discover promising architectures of deep neural networks even in very large search spaces. Despite its success, DARTS lacks robustness in certain cases, e.g. it may degenerate to trivial architectures with excessive parametric-free operations such as skip connection or random noise, leading to inferior performance. In particular, operation selection based on the magnitude of architectural parameters was recently proven to be fundamentally wrong showcasing the need to rethink this aspect. On the other hand, zero-cost proxies have been recently studied in the context of sample-based NAS showing promising results -speeding up the search process drastically in some cases but also failing on some of the large search spaces typical for differentiable NAS.In this work we propose a novel operation selection paradigm in the context of differentiable NAS which utilises zero-cost proxies. Our "perturbation-based zerocost operation selection" (Zero-Cost-PT) improves searching time and, in many cases, accuracy compared to the best available differentiable architecture search, regardless of the search space size. Specifically, we are able to find comparable architectures to DARTS-PT on the DARTS CNN search space while being over 40× faster (total searching time 25 minutes on a single GPU). Our code will be available at: https://github.com/avail-upon-acceptance.

$β$-DARTS++: Bi-level Regularization for Proxy-robust Differentiable Architecture Search

“…NASI [32] utilizes the NTK to characterize the convergence performance of candidate networks, so as to achieve fast search without training. Zero-Cost-PT [33] proposes a training-free proxy, namely perturbation-based operation scoring, to evaluate the performance of subnetworks efficiently. In a word, these methods mainly focus on finding relevant measurements to indicate the quality of architectures, and still have a performance gap with well-trained NAS.…”