MiLeNAS: Efficient Neural Architecture Search via Mixed-Level Reformulation

He, Chaoyang; Ye, Haishan; Shen, Li; Zhang, Tong

doi:10.1109/cvpr42600.2020.01201

Cited by 108 publications

(76 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The approaches to personalization discussed in Section 3.3 still share the same model architecture among all clients. The recent progress in NAS [230,387,175,388,60,375,313,488,175,323] provides a potential way to address these drawbacks. There are three major methods for NAS, which utilize evolutionary algorithms, reinforcement learning, or gradient descent to search for optimal architectures for a specific task on a specific dataset.…”

Section: Neural Architecture Designmentioning

confidence: 99%

Advances and Open Problems in Federated Learning

Kairouz

McMahan²

2021

FNT in Machine Learning

Self Cite

2,515

1,214

View full text Add to dashboard Cite

Federated learning (FL) is a machine learning setting where many clients (e.g. mobile devices or whole organizations) collaboratively train a model under the orchestration of a central server (e.g. service provider), while keeping the training data decentralized. FL embodies the principles of focused data collection and minimization, and can mitigate many of the systemic privacy risks and costs resulting from traditional, centralized machine learning and data science approaches. Motivated by the explosive growth in FL research, this paper discusses recent advances and presents an extensive collection of open problems and challenges.

show abstract

Section: Neural Architecture Designmentioning

confidence: 99%

Advances and Open Problems in Federated Learning

Kairouz

McMahan²

2021

FNT in Machine Learning

Self Cite

2,515

1,214

View full text Add to dashboard Cite

show abstract

“…Most recently GD-based NAS methods are formulated as bilevel optimization problems, However, He et al [116] observe that bilevel optimization in the current methods is solved based on a heuristic. For instance, solution of the problem needs to get an approximation of the second-order methods [109,110].…”

Section: Nas Based On Gdmentioning

confidence: 99%

“…For instance, solution of the problem needs to get an approximation of the second-order methods [109,110]. He et al [116] demonstrate that the approximation has a superposition influence mainly because it is based on a one-step approximation of the network weights. As a result, gradient errors may cause the algorithm to fail to converge to a (locally) optimal solution.…”

Section: Nas Based On Gdmentioning

confidence: 99%

From federated learning to federated neural architecture search: a survey

Zhu

Zhang

Jin

2021

Complex Intell. Syst.

129

View full text Add to dashboard Cite

Federated learning is a recently proposed distributed machine learning paradigm for privacy preservation, which has found a wide range of applications where data privacy is of primary concern. Meanwhile, neural architecture search has become very popular in deep learning for automatically tuning the architecture and hyperparameters of deep neural networks. While both federated learning and neural architecture search are faced with many open challenges, searching for optimized neural architectures in the federated learning framework is particularly demanding. This survey paper starts with a brief introduction to federated learning, including both horizontal, vertical, and hybrid federated learning. Then neural architecture search approaches based on reinforcement learning, evolutionary algorithms and gradient-based are presented. This is followed by a description of federated neural architecture search that has recently been proposed, which is categorized into online and offline implementations, and single- and multi-objective search approaches. Finally, remaining open research questions are outlined and promising research topics are suggested.

show abstract

“…However, single-level optimization will overfit the architecture A and result in performance degradation during the validation process. Then, a mixed-level optimization [68] is proposed to deal with this problem and reduce gradient errors; that is, min w,A…”

Section: Stability and Convergence Analysismentioning

confidence: 99%

Gradient Descent Effects on Differential Neural Architecture Search: A Survey

2021

View full text Add to dashboard Cite

Gradient Descent, an effective way to search for the local minimum of a function, can minimize training and validation loss of neural architectures and also be incited in an appropriate order to decrease the searching cost of neural architecture search. In recent trends, the neural architecture search (NAS) is enormously used to construct an automatic architecture for a specific task. Mostly well-performed neural architecture search methods have adopted reinforcement learning, evolutionary algorithms, or gradient descent algorithms to find the best-performing candidate architecture. Among these methods, gradient descent-based architecture search approaches outperform all other methods in terms of efficiency, simplicity, computational cost, and validation error. In view of this, an in-depth survey is necessary to cover the usefulness of gradient descent method and how this can benefit neural architecture search. We begin our survey with basic concepts of neural architecture search, gradient descent, and their unique properties. Our survey then delves into the impact of gradient descent method on NAS and explores the effect of gradient descent in the search process to generate the candidate architecture. At the same time, our survey reviews mostly used gradient-based search approaches in NAS. Finally, we provide the current research challenges and open problems in the NAS-based approaches, which need to be addressed in future research.

show abstract

MiLeNAS: Efficient Neural Architecture Search via Mixed-Level Reformulation

Cited by 108 publications

References 12 publications

Advances and Open Problems in Federated Learning

Advances and Open Problems in Federated Learning

From federated learning to federated neural architecture search: a survey

Gradient Descent Effects on Differential Neural Architecture Search: A Survey

Contact Info

Product

Resources

About