Progressive Neural Architecture Search

Liu, Chenxi; Zoph, Barret; Neumann, Maxim; Shlens, Jonathon; Wei, Hua; Li, Lijia; Li, Feifei; Yuille, Alan; Huang, Jonathan; Murphy, Kevin

doi:10.48550/arxiv.1712.00559

Cited by 101 publications

(156 citation statements)

References 18 publications

Supporting

Mentioning

154

Contrasting

Order By: Relevance

“…For image classification tasks, most works define a search space in terms of cells, computational graphs of neural primitives (e.g., convolution or pooling operations) that can then be stacked to compose global networks (Zoph and Le, 2016;Liu et al, 2017a). See Figure 1a for an illustration.…”

Section: Related Workmentioning

confidence: 99%

“…In recent years, there has been a surge of interest in automatically identifying neural network architectures, effectively replacing the expense of human time with the expense of computational time. These approaches have often reached or exceeded the level of accuracy obtained by architectures that were tuned manually (Zoph et al, 2017;Cai et al, 2018b;Liu et al, 2017a;Zhong et al, 2018;Zoph and Le, 2016). Most of the early work in this area was focused on defining the search space (e.g.…”

Section: Introductionmentioning

confidence: 99%

“…Architecture surrogates are small versions of the final network that are cheaper to store in memory and faster to train. For example, in image classification, this can be achieved by searching over the architecture of a recurrent convolutional unit, the cell (Liu et al, 2017a;Real et al, 2018;Cai et al, 2018b;Liu et al, 2017c;Tan et al, 2018;Luo et al, 2018), rather than directly on the global architecture. Global networks with different complexities can then be obtained by stacking cells while varying the number of stacks, the number of filters, etc.…”

Section: Introductionmentioning

confidence: 99%

“…For example, one could perform architecture search on CIFAR-10 and "transfer" the resulting architecture (with some modifications, e.g. changing the number of filters) to ImageNet (Liu et al, 2017a;Xie et al, 2019). While good performance on a surrogate task does not guarantee good performance on the final task (i.e.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Probabilistic Neural Architecture Search

Casale¹,

Gordon²,

Fusi³

2019

Preprint

View full text Add to dashboard Cite

In neural architecture search (NAS), the space of neural network architectures is automatically explored to maximize predictive accuracy for a given task. Despite the success of recent approaches, most existing methods cannot be directly applied to large scale problems because of their prohibitive computational complexity or high memory usage. In this work, we propose a Probabilistic approach to neural ARchitecture SEarCh (PARSEC) that drastically reduces memory requirements while maintaining state-of-the-art computational complexity, making it possible to directly search over more complex architectures and larger datasets. Our approach only requires as much memory as is needed to train a single architecture from our search space. This is due to a memory-efficient sampling procedure wherein we learn a probability distribution over high-performing neural network architectures. Importantly, this framework enables us to transfer the distribution of architectures learnt on smaller problems to larger ones, further reducing the computational cost. We showcase the advantages of our approach in applications to CIFAR-10 and ImageNet, where our approach outperforms methods with double its computational cost and matches the performance of methods with costs that are three orders of magnitude larger.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Probabilistic Neural Architecture Search

Casale¹,

Gordon²,

Fusi³

2019

Preprint

View full text Add to dashboard Cite

show abstract

“…There are too many factors to consider, which is a huge challenge to design an efficient model. To automate the architecture design process, RL was first introduced to search for efficient architectures with competitive accuracy [201][202][203][204][205]. A fully configurable search space can grow exponentially large and intractable.…”

mentioning

confidence: 99%

Review of Image Classification Algorithms Based on Convolutional Neural Networks

et al. 2021

View full text Add to dashboard Cite

Image classification has always been a hot research direction in the world, and the emergence of deep learning has promoted the development of this field. Convolutional neural networks (CNNs) have gradually become the mainstream algorithm for image classification since 2012, and the CNN architecture applied to other visual recognition tasks (such as object detection, object localization, and semantic segmentation) is generally derived from the network architecture in image classification. In the wake of these successes, CNN-based methods have emerged in remote sensing image scene classification and achieved advanced classification accuracy. In this review, which focuses on the application of CNNs to image classification tasks, we cover their development, from their predecessors up to recent state-of-the-art (SOAT) network architectures. Along the way, we analyze (1) the basic structure of artificial neural networks (ANNs) and the basic network layers of CNNs, (2) the classic predecessor network models, (3) the recent SOAT network algorithms, (4) comprehensive comparison of various image classification methods mentioned in this article. Finally, we have also summarized the main analysis and discussion in this article, as well as introduce some of the current trends.

show abstract

DPP-Net: Device-Aware Progressive Search for Pareto-Optimal Neural Architectures

Dong

Cheng

Juan

et al. 2018

Lecture Notes in Computer Science

175

159

View full text Add to dashboard Cite

Recent breakthroughs in Neural Architectural Search (NAS) have achieved state-of-the-art performances in applications such as image classification and language modeling. However, these techniques typically ignore device-related objectives such as inference time, memory usage, and power consumption. Optimizing neural architecture for devicerelated objectives is immensely crucial for deploying deep networks on portable devices with limited computing resources. We propose DPP-Net: Device-aware Progressive Search for Pareto-optimal Neural Architectures, optimizing for both device-related (e.g., inference time and memory usage) and device-agnostic (e.g., accuracy and model size) objectives. DPP-Net employs a compact search space inspired by current state-of-the-art mobile CNNs, and further improves search efficiency by adopting progressive search . Experimental results on CIFAR-10 are poised to demonstrate the effectiveness of Pareto-optimal networks found by DPP-Net, for three different devices: (1) a workstation with Titan X GPU, (2) NVIDIA Jetson TX1 embedded system, and (3) mobile phone with ARM Cortex-A53. Compared to CondenseNet and NASNet (Mobile), DPP-Net achieves better performances: higher accuracy & shorter inference time on various devices. Additional experimental results show that models found by DPP-Net also achieve considerablygood performance on ImageNet as well.

show abstract

Progressive Neural Architecture Search

Cited by 101 publications

References 18 publications

Probabilistic Neural Architecture Search

Probabilistic Neural Architecture Search

Review of Image Classification Algorithms Based on Convolutional Neural Networks

DPP-Net: Device-Aware Progressive Search for Pareto-Optimal Neural Architectures

Contact Info

Product

Resources

About