RARTS: An Efficient First-Order Relaxed Architecture Search Method

Abstract

Differentiable architecture search (DARTS) is an effective method for data-driven neural network design based on solving a bilevel optimization problem. Despite its success in many architecture search tasks, there are still some concerns about the accuracy of first-order DARTS and the efficiency of the second-order DARTS. In this paper, we formulate a single level alternative and a relaxed architecture search (RARTS) method that utilizes the whole dataset in architecture learning via both data and network splitting, without involving mixed second derivatives of the corresponding loss functions like DARTS. In our formulation of network splitting, two networks with different but related weights cooperate in search of a shared architecture. The advantage of RARTS over DARTS is justified by a convergence theorem and an analytically solvable model. Moreover, RARTS outperforms DARTS and its variants in accuracy and search efficiency, as shown in adequate experimental results. For the task of searching topological architecture, i.e., the edges and the operations, RARTS obtains a higher accuracy and 60% reduction of computational cost than second-order DARTS on CIFAR-10. RARTS continues to out-perform DARTS upon transfer to ImageNet and is on par with recent variants of DARTS even though our innovation is purely on the training algorithm without modifying search space. For the task of searching width, i.e., the number of channels in convolutional layers, RARTS also outperforms the traditional network pruning benchmarks. Further experiments on the public architecture search benchmark like NATS-Bench also support the preeminence of RARTS.INDEX TERMS Convolutional neural networks, neural architecture search, differentiable architecture search, network compression.

Systematic Review on Neural Architecture Search

“…In order to have continuous SSp and enable GO, all connections in architecture are assigned a variable. Then, the SSt optimizes the performance of the architecture by changing those variables that define the state of the connection (appearance in the network or not) at the end of the optimization [87,88]. As was shown in Figure 9, only the connections with the highest values were retained, and the others were removed.…”

“…Regularized Differentiable Architecture Search (RDARTS) [89] Fig. 9: A simplified SSp of differentiable architecture search (DARTS) [87]: non-faded connections (or layers) are selected for the architecture, blocks are different data stages, and different colors show different types of layers like CNN, skip, etc. overcomes this problem by utilizing regularization to encourage the RL agent to choose architectures that are capable of being more generalized and less overfitted.…”