Cyclic Differentiable Architecture Search

Yu, Hongyuan; Peng, Houwen; Huang, Yan; Fu, Jianlong; Du, Hao; Wang, Liang; Ling, Haibin

doi:10.1109/tpami.2022.3153065

Cited by 28 publications

(6 citation statements)

References 72 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…GDAS guided the search toward architectures with lower validation loss. Addressing the problem that NAS algorithms are computationally inefficient, CDARTS, which stands for a Cyclic Differentiable Architecture Search [90], developed a two-step optimization to solve the problem. In every iteration of the algorithm, both the architecture and weights of the model are being trained: first, the weights of the model are trained; and then, the hyperparameters are evaluated to optimize the architecture.…”

Section: Methodsmentioning

confidence: 99%

Systematic Review on Neural Architecture Search

Avval,

Yaghoubi,

Eskue

et al. 2024

Preprint

View full text Add to dashboard Cite

Machine Learning (ML) has revolutionized various fields, enabling the development of intelligent systems capable of solving complex problems. However, the process of manually designing and optimizing ML models is often timeconsuming, labor-intensive, and requires specialized expertise. To address these challenges, Automatic Machine Learning (AutoML) has emerged as a promising approach that automates the process of selecting and optimizing ML models. Within the realm of AutoML, Neural Architecture Search (NAS) has emerged as a powerful technique that automates the design of neural network architectures, the core components of ML models. It has recently gained significant attraction due to its capability to discover novel and efficient architectures that surpass human-designed counterparts. This manuscript aims to present a systematic review of the literature on this topic published between 2017 and 2023 to identify, analyze, and classify the different types of algorithms developed for NAS. The methodology follows the guidelines of Systematic Literature Review (SLR) methods. Consequently, this study identified 160 articles that provide a comprehensive overview of the field of NAS, encompassing discussion on current works, their purposes, conclusions, and predictions of the direction of this science branch in its main core pillars: Search Space (SSp), Search Strategy (SSt), and Validation Strategy (VSt). Subsequently, the key milestones and advancements that have shaped the field are highlighted. Moreover, we discuss the challenges and open issues that remain in the field. We envision that NAS will continue to play a pivotal role in the advancement of ML, enabling the development of more intelligent and efficient ML models for a wide range of applications.

show abstract

Section: Methodsmentioning

confidence: 99%

Systematic Review on Neural Architecture Search

Avval,

Yaghoubi,

Eskue

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…He et al [54] proposed a mix-level objective function and also used a model size-based search strategy combined with an early stopping strategy, successfully improved the performance of the final architecture. Different improvements have also been made explicitly by other researchers to alleviate the problems present in DARTS [55][56][57][58].…”

Section: Related Workmentioning

confidence: 99%

“…However, the use of these methods that are explicit, such as an early stopping strategy, incorporating an attention mechanism, or modifying the composition of the search space, all add additional artificially set hyper-parameters. Therefore there are also many implicit approaches to increase the stability of architectural search, such as using architectural parameter regularization.PC-DARTS [44], FairDARTS [50], DOTS [53], CDARTS [55],…”

Section: Relationship With Previous Workmentioning

confidence: 99%

“…Explicit P-DARTS [45] Adopting search space regularization ASM-NAS [47] Incorporating attention mechanisms in architectural search Att-DARTS [49] Applying attention mechanisms to search space DARTS+ [51] Adopting an early stopping strategy DARTS- [52] Introducing an auxiliary skip connection MileNAS [54] Adoption of early stopping strategy and mixed-level objective function R-DARTS [59] Using early stopping strategy and L2 regularization SharpDARTS [61] Using HyperCuboid search space and Max-W weight regularization ADARTS [65] Employing attention-based partial channel connection strategy Implicit PC-DARTS [44] Using coefficient regularization FairDARTS [50] Using the 0-1 loss function DOTS [53] Using decoupled operations and topology search CDARTS [55] Introducing an introspective distillation mechanism MS-DARTS [27] Using mean shift architecture parameter regularization GOLD-NAS [28] Leveraging computational resources to regularize architectural parameters GAEA-DARTS [29] Updating architecture parameters using exponential gradients β-DARTS [30] Using softmax function architecture parameter regularization InLM-NAS Introducing inner-loop mechanism regularization and gradient modification problem. In addition, the approximation used in this paper is more accurate compared to the one-step approximation used by all the above methods on the objective function.…”

Section: Table ⅰ Comparison With Prior Workmentioning

confidence: 99%

See 1 more Smart Citation

Inner Loop-Based Modified Differentiable Architecture Search

Jin,

Huang

2024

IEEE Access

View full text Add to dashboard Cite

Differentiable neural architecture search, which significantly reduces the computational cost of architecture search by several orders of magnitude, has become a popular research issue in recent years. Architecture search can fundamentally be described as an optimization problem. The differentiable architecture search updates the search process based on gradients, then derives the final sub-network architecture from the super network of the search space. However, the gap between the super network and its sub-networks together with the inaccuracy of the gradient approximation during architecture optimization bring performance collapse problems in the architecture search, making the search process extremely unstable. To this end, we propose an inner loop-based modified differentiable neural architecture search method (InLM-NAS). Firstly, we redefine the objective function of the architecture optimization process in the search process by introducing an inner-loop mechanism to prevent overfitting problems of architecture parameters and avoid convergence of the architecture search to suboptimal architectures. Secondly, a novel approximation calculation is introduced in the architecture optimization process, which reduces the error caused by the gradient approximation. It alleviates the sensitivity to the hyper-parameters setting during the architecture search and enhances the stability of the architecture search. Finally, extensive validation experiments on public datasets demonstrate that our proposed method has a more robust search process, and the searched neural network architecture has a superior network performance.

show abstract

“…Chen et al 19 proposed the idea of the progressive search strategy, which alleviates the adverse effects of the depth gap problem to some extent. Many different search strategies 20 – 23 have also been proposed to alleviate the adverse effects of the depth gap problem. In addition, the attention mechanism can help the neural network select useful features and discard the less-useful ones.…”

Section: Introductionmentioning

confidence: 99%

Neural architecture search via progressive partial connection with attention mechanism

Jin,

Huang,

Chen

2024

Sci Rep

View full text Add to dashboard Cite

Differentiable architecture search requires a larger computational consumption during architecture search, and there exists the depth gap problem under deeper network architecture. In this paper, we propose an attention-based progressive partially connected neural architecture search method (PPCAtt-NAS) to address these two issues. First, we introduce a progressive search strategy in the architecture search phase, build up the sophistication of the architecture gradually and perform path-level pruning in stages to bridge the depth gap. Second, we adopt a partial search scheme that performs channel-level partial sampling of the network architecture to further reduce the computational complexity of the architecture search. In addition, an attention mechanism is devised to improve the architecture search capability by enhancing the relevance between the feature channels. Finally, we conduct extensive comparison experiments with state-of-the-art methods on several public datasets, and our method is able to present higher architecture performance.

show abstract

Cyclic Differentiable Architecture Search

Cited by 28 publications

References 72 publications

Systematic Review on Neural Architecture Search

Systematic Review on Neural Architecture Search

Inner Loop-Based Modified Differentiable Architecture Search

Neural architecture search via progressive partial connection with attention mechanism

Contact Info

Product

Resources

About