TextNAS: A Neural Architecture Search Space Tailored for Text Representation

Abstract

Learning text representation is crucial for text classification and other language related tasks. There are a diverse set of text representation networks in the literature, and how to find the optimal one is a non-trivial problem. Recently, the emerging Neural Architecture Search (NAS) techniques have demonstrated good potential to solve the problem. Nevertheless, most of the existing works of NAS focus on the search algorithms and pay little attention to the search space. In this paper, we argue that the search space is also an important human prior to the success of NAS in different applications. Thus, we propose a novel search space tailored for text representation. Through automatic search, the discovered network architecture outperforms state-of-the-art models on various public datasets on text classification and natural language inference tasks. Furthermore, some of the design principles found in the automatic network agree well with human intuition.

AutoADR

“…Once the supernet is trained, each sampled structure can be directly and quickly evaluated without training from scratch. In our work, we also leverage the weight sharing strategy and build a one-shot model [8] with TextNAS [22] search space, while random search is adopted to sample architectures. Besides, we incorporate knowledge distillation and efficiency constraints as search hints to obtain effective and light student models.…”

“…Neural Architecture Search: The task is to find an effective and efficient architecture for AutoADR sub-model through teacherstudent framework. We adopt the search space in TextNAS [22] and build a corresponding network subsuming all possible architectures, which is called supernet [8] for the one-shot search algorithm. To apply knowledge distillation, we train the supernet with uniform sampling under the guidance of the soft predictions from teacher model.…”

“…In our experiments, we set = 6 as a trade-off between expressiveness and efficiency. Following TextNAS [22], the network architecture of the encoder supports multi-path ensemble, which is a common design principle of manual networks.…”

“…EfficientNet [19] proposed a simple and highly effective compound scaling method that enables easy scaling up of a backbone model and beats the performances of human-designed image recognition architectures by a large margin. In the NLP domain, some of us proposed TextNAS [22], which designed a novel search space tailored for text representation and achieved state-of-the-art performances on various natural language processing tasks when pre-training was not applied.…”

“…The NAS procedure is performed jointly with knowledge distillation in an iterative manner, so that NAS and knowledge distillation will collaborate with each other to find a tailored model architecture to achieve our goals on both accuracy and efficiency. Moreover, following TextNAS [22], we leverage a customized search space for text representation, which consists of a multi-path mixture of convolutional, recurrent, pooling, and self-attention layers. In this way, we can explore the best composition of different layers for the Ad Relevance prediction problem.…”