Learning Architectures from an Extended Search Space for Language Modeling

Abstract

Neural architecture search (NAS) has advanced significantly in recent years but most NAS systems restrict search to learning architectures of a recurrent or convolutional cell. In this paper, we extend the search space of NAS. In particular, we present a general approach to learn both intra-cell and inter-cell architectures (call it ESS). For a better search result, we design a joint learning method to perform intra-cell and inter-cell NAS simultaneously. We implement our model in a differentiable architecture search system. For recurrent neural language modeling, it outperforms a strong baseline significantly on the PTB and Wiki-Text data, with a new state-of-the-art on PTB. Moreover, the learned architectures show good transferability to other systems. E.g., they improve state-of-the-art systems on the CoNLL and WNUT named entity recognition (NER) tasks and CoNLL chunking task, indicating a promising line of research on large-scale prelearned architectures.

Compositional Demographic Word Embeddings

“…A range of models have been developed, with progressively larger models trained on more data (e.g., Dai et al, 2019). Variations of the LSTM have consistently achieved state-of-the-art performance without massive compute resources (Merity et al, 2018a;Melis et al, 2019;Merity, 2019;Li et al, 2020). We use the AWD-LSTM (Merity et al, 2018b) in our experiments, as it achieves very strong performance, has a well-studied codebase, and can be trained on a single GPU in a day.…”

Improving Low Compute Language Modeling with In-Domain Embedding Initialisation

“…We focus on a single model type for computational budget reasons. We chose an LSTM because while transformer based models such as GPT-2 now dominate transfer learning, LSTMs continue to be competitive in language modeling (Du et al, 2020;Li et al, 2020;Melis et al, 2018;Merity et al, 2017a). Our ideas are orthogonal to this prior work and our findings may apply to transformers as well, but confirming that would require additional experiments.…”

Evaluating the Effectiveness of Efficient Neural Architecture Search for Sentence-Pair Tasks

“…NAS methods have shown strong performance on many NLP and CV tasks, such as language model-ing and image classification (Zoph and Le, 2017;Pham et al, 2018;Luo et al, 2018;Liu et al, 2019). Applications in NLP, such as NER (Jiang et al, 2019;Li et al, 2020), translation (So et al, 2019), text classification (Wang et al, 2020), and natural language inference (NLI) (Pasunuru and Bansal, 2019;Wang et al, 2020) have also been explored.…”

“…Current SOTA approaches focus on learning new cell architectures as replacements for LSTM or convolutional cells (Zoph and Le, 2017;Pham et al, 2018;Liu et al, 2019;Jiang et al, 2019;Li et al, 2020) or entire model architectures to replace hand-designed models such as the transformer or DenseNet (So et al, 2019;Pham et al, 2018).…”