Local Search is a Remarkably Strong Baseline for Neural Architecture Search

Ottelander, T. Den; Dushatskiy, Arkadiy; Virgolin, Marco; Bosman, Peter A. N.

doi:10.1007/978-3-030-72062-9_37

Cited by 24 publications

(29 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…NAS algorithms. For single-fidelity algorithms, we implemented random search (RS) [34], local search (LS) [68,48], regularized evolution (REA) [53], and BANANAS [67]. For multi-fidelity bandit-based algorithms, we implemented Hyperband (HB) [35] and Bayesian optimization Hyperband (BOHB) [14].…”

Section: Methodsmentioning

confidence: 99%

“…Now we describe a simple framework for converting single-fidelity NAS algorithms to multi-fidelity NAS algorithms using learning curve extrapolation techniques. We show that this framework is able to substantially improve the performance of popular algorithms such as regularized evolution [53], BANANAS [67], and local search [68,48].…”

Section: The Power Of Learning Curve Extrapolationmentioning

confidence: 99%

See 1 more Smart Citation

NAS-Bench-x11 and the Power of Learning Curves

Shen¹,

White²,

Savani³

et al. 2021

Preprint

View full text Add to dashboard Cite

While early research in neural architecture search (NAS) required extreme computational resources, the recent releases of tabular and surrogate benchmarks have greatly increased the speed and reproducibility of NAS research. However, two of the most popular benchmarks do not provide the full training information for each architecture. As a result, on these benchmarks it is not possible to run many types of multi-fidelity techniques, such as learning curve extrapolation, that require evaluating architectures at arbitrary epochs. In this work, we present a method using singular value decomposition and noise modeling to create surrogate benchmarks, NAS-Bench-111, NAS-Bench-311, and NAS-Bench-NLP11, that output the full training information for each architecture, rather than just the final validation accuracy. We demonstrate the power of using the full training information by introducing a learning curve extrapolation framework to modify single-fidelity algorithms, showing that it leads to improvements over popular single-fidelity algorithms which claimed to be state-of-the-art upon release. Our code and pretrained models are available at https://github.com/automl/nas-bench-x11.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: The Power Of Learning Curve Extrapolationmentioning

confidence: 99%

NAS-Bench-x11 and the Power of Learning Curves

Shen¹,

White²,

Savani³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…A black-box NAS algorithm is an algorithm which iteratively chooses architectures to train, and then uses the final validation accuracies in the next iteration. We run experiments for five popular black-box NAS algorithms: random search (RS) (Li & Talwalkar, 2019;Sciuto et al, 2020), regularized evolution (RE) , local search (LS) (White et al, 2021b;Ottelander et al, 2021), BANANAS (White et al, 2021a), and NPENAS . We run each black-box algorithm for 200 iterations.…”

Section: On the Generalizability Of Nas Algorithmsmentioning

confidence: 99%

“…• Local search. Another baseline, local search has been shown to perform well on multiple NAS benchmarks (White et al, 2021b;Ottelander et al, 2021;Siems et al, 2020). It works by evaluating all architectures in the neighborhood of the current best architecture found so far.…”

Section: Details From Section 4 C1 Nas Algorithm Implementation Detailsmentioning

confidence: 99%

“…In this work, we show that there is a need to adopt newer benchmarks because many conclusions drawn from a small subset of benchmarks do not generalize across diverse datasets and tasks. Specifically, we present an in-depth analysis of popular black-box White et al, 2021a;Ottelander et al, 2021), one-shot (Liu et al, 2019b;Dong & Yang, 2019), and performance prediction methods (White et al, 2021c) across (nearly) every publicly available queryable NAS benchmark. This includes 25 different combinations of search spaces and datasets, which is, to the best of our knowledge, by far the largest set of NAS search spaces and datasets on which experiments have been conducted to date.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

NAS-Bench-Suite: NAS Evaluation is (Now) Surprisingly Easy

Mehta¹,

White²,

Zela³

et al. 2022

Preprint

View full text Add to dashboard Cite

The release of tabular benchmarks, such as NAS-Bench-101 and NAS-Bench-201, has significantly lowered the computational overhead for conducting scientific research in neural architecture search (NAS). Although they have been widely adopted and used to tune real-world NAS algorithms, these benchmarks are limited to small search spaces and focus solely on image classification. Recently, several new NAS benchmarks have been introduced that cover significantly larger search spaces over a wide range of tasks, including object detection, speech recognition, and natural language processing. However, substantial differences among these NAS benchmarks have so far prevented their widespread adoption, limiting researchers to using just a few benchmarks. In this work, we present an in-depth analysis of popular NAS algorithms and performance prediction methods across 25 different combinations of search spaces and datasets, finding that many conclusions drawn from a few NAS benchmarks do not generalize to other benchmarks. To help remedy this problem, we introduce NAS-Bench-Suite, a comprehensive and extensible collection of NAS benchmarks, accessible through a unified interface, created with the aim to facilitate reproducible, generalizable, and rapid NAS research. Our code is available at https://github.com/automl/naslib.

show abstract