Approximate Neural Architecture Search via Operation Distribution Learning

Wan, Xingchen; Ru, Binxin; Esperança, Pedro M.; Carlucci, Fabio Maria

doi:10.1109/wacv51458.2022.00360

Cited by 8 publications

(8 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We can observe that both the normal cell and the reduction cell have the skip connect operation from the input nodes, leading to a residual link with other operations. As pointed out in [44], such a ResNet-style residual link is helpful for achieving state-of-the-art performance, demonstrating that our method excels in identifying excellent architectures.…”

Section: Resultsmentioning

confidence: 60%

PA&DA: Jointly Sampling PAth and DAta for Consistent NAS

Lü¹,

Yu²,

Yang³

et al. 2023

Preprint

View full text Add to dashboard Cite

Based on the weight-sharing mechanism, one-shot NAS methods train a supernet and then inherit the pre-trained weights to evaluate sub-models, largely reducing the search cost. However, several works have pointed out that the shared weights suffer from different gradient descent directions during training. And we further find that large gradient variance occurs during supernet training, which degrades the supernet ranking consistency. To mitigate this issue, we propose to explicitly minimize the gradient variance of the supernet training by jointly optimizing the sampling distributions of PAth and DAta (PA&DA). We theoretically derive the relationship between the gradient variance and the sampling distributions, and reveal that the optimal sampling probability is proportional to the normalized gradient norm of path and training data. Hence, we use the normalized gradient norm as the importance indicator for path and training data, and adopt an importance sampling strategy for the supernet training. Our method only requires negligible computation cost for optimizing the sampling distributions of path and data, but achieves lower gradient variance during supernet training and better generalization performance for the supernet, resulting in a more consistent NAS. We conduct comprehensive comparisons with other improved approaches in various search spaces. Results show that our method surpasses others with more reliable ranking performance and higher accuracy of searched architectures, showing the effectiveness of our method. Code is available at https://github.com/ShunLu91/PA-DA.

show abstract

Section: Resultsmentioning

confidence: 60%

PA&DA: Jointly Sampling PAth and DAta for Consistent NAS

Lü¹,

Yu²,

Yang³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…In this work, we set out to evaluate the impact that the operations, their combination, and their occurrences have in the final validation accuracy of an architecture in NAS-Bench-201, as well as evaluating the search space distribution in terms of the architecture's accuracy and their correlation between data sets. With this, we extend prior works that looked into proposing best practices for NAS methods, as well as evaluating search spaces and benchmarks [32], [42], [54], [62].…”

Section: Introductionmentioning

confidence: 91%

“…conducted an extensive study to evaluate the impact of the training protocols on the results of a generated architecture, showing that wellengineered training protocols usually have a greater impact on the final performance of a NAS method than the search arXiv:2303.16938v1 [cs.LG] 29 Mar 2023 strategy, thus suggesting that NAS methods should provide results using the same protocols to ensure fair comparisons [62]. In [54], the authors analyzed popular cell-based search spaces. They found that existing search spaces contain a high degree of redundancy and generated architectures from distinct NAS methods have similar patterns.…”

Section: Introductionmentioning

confidence: 99%

Efficient guided evolution for neural architecture search

Lopes

Santos

Degardin

et al. 2022

Proceedings of the Genetic and Evolutionary Computation Conference Companion

View full text Add to dashboard Cite

Neural Architecture Search (NAS) benchmarks significantly improved the capability of developing and comparing NAS methods while at the same time drastically reduced the computational overhead by providing meta-information about thousands of trained neural networks. However, tabular benchmarks have several drawbacks that can hinder fair comparisons and provide unreliable results. These usually focus on providing a small pool of operations in heavily constrained search spaces -usually cell-based neural networks with pre-defined outerskeletons. In this work, we conducted an empirical analysis of the widely used NAS-Bench-101, NAS-Bench-201 and TransNAS-Bench-101 benchmarks in terms of their generability and how different operations influence the performance of the generated architectures. We found that only a subset of the operation pool is required to generate architectures close to the upper-bound of the performance range. Also, the performance distribution is negatively skewed, having a higher density of architectures in the upper-bound range. We consistently found convolution layers to have the highest impact on the architecture's performance, and that specific combination of operations favors top-scoring architectures. These findings shed insights on the correct evaluation and comparison of NAS methods using NAS benchmarks, showing that directly searching on NAS-Bench-201, ImageNet16-120 and TransNAS-Bench-101 produces more reliable results than searching only on CIFAR-10. Furthermore, with this work we provide suggestions for future benchmark evaluations and design. The code used to conduct the evaluations is available at https://github.com/VascoLopes/NAS-Benchmark-Evaluation.

show abstract

“…This clearly shows that the search space design can be further optimised, but consequently, many beliefs often ingrained in existing search methods can also be sub-optimal or unnecessary. For example, barring a few exceptions (Xie et al, 2019a;You et al, 2020;Ru et al, 2020;Wan et al, 2022), the overwhelming majority of the current approaches aims to search for a single, fully deterministic architecture, and this often results in high-dimensional vector encoding of the cells (e.g. the path encoding of DARTS cell in White et al (2021) is > 10 4 dimensions without truncation).…”

Section: Normalmentioning

confidence: 99%

On Redundancy and Diversity in Cell-based Neural Architecture Search

Wan¹,

Ru²,

Esperança³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Searching for the architecture cells is a dominant paradigm in NAS. However, little attention has been devoted to the analysis of the cell-based search spaces even though it is highly important for the continual development of NAS. In this work, we conduct an empirical post-hoc analysis of architectures from the popular cellbased search spaces and find that the existing search spaces contain a high degree of redundancy: the architecture performance is minimally sensitive to changes at large parts of the cells, and universally adopted designs, like the explicit search for a reduction cell, significantly increase the complexities but have very limited impact on the performance. Across architectures found by a diverse set of search strategies, we consistently find that the parts of the cells that do matter for architecture performance often follow similar and simple patterns. By explicitly constraining cells to include these patterns, randomly sampled architectures can match or even outperform the state of the art. These findings cast doubts into our ability to discover truly novel architectures in the existing cell-based search spaces, and inspire our suggestions for improvement to guide future NAS research. Code is available at https: //github.com/xingchenwan/cell-based-NAS-analysis.

show abstract

Approximate Neural Architecture Search via Operation Distribution Learning

Cited by 8 publications

References 13 publications

PA&DA: Jointly Sampling PAth and DAta for Consistent NAS

PA&DA: Jointly Sampling PAth and DAta for Consistent NAS

Efficient guided evolution for neural architecture search

On Redundancy and Diversity in Cell-based Neural Architecture Search

Contact Info

Product

Resources

About