NAS-Bench-NLP: Neural Architecture Search Benchmark for Natural Language Processing

Abstract

Neural Architecture Search (NAS) is a promising and rapidly evolving research area. Training a large number of neural networks requires an exceptional amount of computational power, which makes NAS unreachable for those researchers who have limited or no access to high-performance clusters and supercomputers. A few benchmarks with precomputed neural architectures performances have been recently introduced to overcome this problem and ensure reproducible experiments. However, these benchmarks are only for the computer vision domain and, thus, are built from the image datasets and convolution-derived architectures. In this work, we step outside the computer vision domain by leveraging the language modeling task, which is the core of natural language processing (NLP). Our main contribution is as follows: we have provided search space of recurrent neural networks on the text datasets and trained 14k architectures within it; we have conducted both intrinsic and extrinsic evaluation of the trained models using datasets for semantic relatedness and language understanding evaluation; finally, we have tested several NAS algorithms to demonstrate how the precomputed results can be utilized. We consider that the benchmark will provide more reliable empirical findings in the community and stimulate progress in developing new NAS methods well suited for recurrent architectures.INDEX TERMS Benchmark, natural language processing, neural architecture search, recurrent neural network.

Exploring robust architectures for deep artificial neural networks

“…However, there are a plethora of other applications and tasks where robustness quantification of DANNs is important [51][52][53][54] . A possible future direction is to extend the presented analysis to more complex applications (e.g., natural language processing, graph data, and Deep Chip industry [55][56][57][58][59] ) and larger models (e.g., Transformers, and Vision Transformers 60,61 ). Given our current analysis, we anticipate that for the larger datasets, complex tasks, and huge models, the graph robustness measures will be even more relevant and will help users/autoML/NAS algorithms find robust DANN architectures.…”

Speeding up deep neural architecture search for wearable activity recognition with early prediction of converged performance

“…This highlights a problem also discussed in Elsken et al (2019b), namely that there are more factors than the architecture of a network which affect it's performance. This indicates a need for a common benchmark for NAS on HAR datasets, following the examples of Dong and Yang (2020) for computer vision and Klyuchnikov et al (2020) for natural language processing, which would allow us to test NAS methods on a search space of pre-trained and pre-evaluated models.…”

Multi-Fidelity Neural Architecture Search With Knowledge Distillation

“…NAS Benchmarks. Several benchmarks [9], [23] have been released to facilitate research on Neural Architecture Search. A NAS Benchmark contains the set a of trained architectures from a search space with a detailed training information.…”