Neural Architecture Search for Convolutional Neural Networks with Attention

Abstract

The recent development of neural architecture search (NAS) has enabled us to automatically discover architectures of neural networks with high performance within a few days. Convolutional neural networks extract fruitful features by repeatedly applying standard operations (convolutions and poolings). However, these operations also extract useless or even disturbing features. Attention mechanisms enable neural networks to discard information of no interest, having achieved the state-of-the-art performance. While a variety of attentions for CNNs have been proposed, current NAS methods have paid a little attention to them. In this study, we propose a novel NAS method that searches attentions as well as operations. We examined several patterns to arrange attentions and operations, and found that attentions work better when they have their own search space and follow operations. We demonstrate the superior performance of our method in experiments on CIFAR-10, CIFAR-100, and ImageNet datasets. The found architecture achieved lower classification error rates and required fewer parameters compared to those found by current NAS methods.

Systematic Review on Neural Architecture Search

“…The downside of this algorithm is its running time; it takes more than 50 GPU days to optimize the model. Moreover, there are different methods, except the mentioned algorithm, in the literature like binarized neural networks [147], swarm intelligence [148], greedy optimizers [47,149], novelty search strategy [150], attention-based search [151], slow-fast learning [152], enhanced RL mixed with a new reward function [60], etc.…”

Inner Loop-Based Modified Differentiable Architecture Search

“…Hao et al [47] proposed to introduce an attention mechanism in the network architecture to help the information interaction between candidate architectures, enabling the search process to focus on selecting better network architectures. Weng et al [48] and Nakai et al [49] both added attention mechanism modules as an operation to the search space. The former added the attention module directly into the search space, while the latter introduced a new attention search space containing multiple attention operations and concatenated the operations searched in the attention search space with the operations searched in the original search space, and both approaches improved the performance of the network architecture.…”

“…Other gradient-based neural network architecture search methods have explicit and implicit approaches to deal with the problem of poor stability in architecture search, respectively. P-DARTS [45], ASM-NAS [47], Att-DARTS [49], DARTS+ [51], DARTS- [52], MileNAS [54], R-DARTS [59],…”

“…Explicit P-DARTS [45] Adopting search space regularization ASM-NAS [47] Incorporating attention mechanisms in architectural search Att-DARTS [49] Applying attention mechanisms to search space DARTS+ [51] Adopting an early stopping strategy DARTS- [52] Introducing an auxiliary skip connection MileNAS [54] Adoption of early stopping strategy and mixed-level objective function R-DARTS [59] Using early stopping strategy and L2 regularization SharpDARTS [61] Using HyperCuboid search space and Max-W weight regularization ADARTS [65] Employing attention-based partial channel connection strategy Implicit PC-DARTS [44] Using coefficient regularization FairDARTS [50] Using the 0-1 loss function DOTS [53] Using decoupled operations and topology search CDARTS [55] Introducing an introspective distillation mechanism MS-DARTS [27] Using mean shift architecture parameter regularization GOLD-NAS [28] Leveraging computational resources to regularize architectural parameters GAEA-DARTS [29] Updating architecture parameters using exponential gradients β-DARTS [30] Using softmax function architecture parameter regularization InLM-NAS Introducing inner-loop mechanism regularization and gradient modification problem. In addition, the approximation used in this paper is more accurate compared to the one-step approximation used by all the above methods on the objective function.…”

Neural architecture search via progressive partial connection with attention mechanism

“…In addition, the attention mechanism can help the neural network select useful features and discard the less-useful ones. Attention mechanism modules have been introduced to enrich the search space 24 , 25 to improve the architecture search performance. Some other methods have also used different search strategies 26 – 30 to try to alleviate the above problems.…”