Evaluating the Effectiveness of Efficient Neural Architecture Search for Sentence-Pair Tasks

Abstract

Neural Architecture Search (NAS) methods, which automatically learn entire neural model or individual neural cell architectures, have recently achieved competitive or state-of-theart (SOTA) performance on variety of natural language processing and computer vision tasks, including language modeling, natural language inference, and image classification. In this work, we explore the applicability of a SOTA NAS algorithm, Efficient Neural Architecture Search (ENAS) (Pham et al., 2018) to two sentence pair tasks, paraphrase detection and semantic textual similarity. We use ENAS to perform a microlevel search and learn a task-optimized RNN cell architecture as a drop-in replacement for an LSTM. We explore the effectiveness of ENAS through experiments on three datasets (MRPC, SICK, STS-B), with two different models (ESIM, BiLSTM-Max), and two sets of embeddings (Glove, BERT). In contrast to prior work applying ENAS to NLP tasks, our results are mixed -we find that ENAS architectures sometimes, but not always, outperform LSTMs and perform similarly to random architecture search.

NAS-Bench-NLP: Neural Architecture Search Benchmark for Natural Language Processing

“…In the NLP domain NAS methods for RNN's have been applied to paraphrase detection [30], named entity recognition [31], language modelling and chunking [32], reaching state-of-the-art performance at the time. TextNAS [33] is another search space, consisting of a mixture of convolutional, recurrent, pooling, and self-attention layers, used for multiple sentence classification and sentence pair modelling tasks.…”