TextNAS: A Neural Architecture Search Space tailored for Text Representation

Wang, Yujing; Yang, Yaming; Chen, Yiren; Bai, Jing; Zhang, Ce; Su, Guinan; Kou, Xiaoyu; Tong, Yunhai; Yang, Mao; Zhou, Lidong

doi:10.48550/arxiv.1912.10729

Cited by 2 publications

(7 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Pham et al [19] propose an extension of NAS, namely ENAS, to speed up training processing by forcing all child networks to share weights. Apart from algorithms, NAS also has many valuable applications such as image classification [20,21], video segmentation [22], text representation [23] and etc. Hence, NAS is a demonstrated powerful tool and it is especially useful in continual learning scenarios when one needs to determine what is a good architecture for the new task.…”

Section: Related Workmentioning

confidence: 99%

Efficient Architecture Search for Continual Learning

Gao¹,

Zhipeng²,

Klabjan³

2020

Preprint

View full text Add to dashboard Cite

Continual learning with neural networks is an important learning framework in AI that aims to learn a sequence of tasks well. However, it is often confronted with three challenges: (1) overcome the catastrophic forgetting problem, (2) adapt the current network to new tasks, and meanwhile (3) control its model complexity. To reach these goals, we propose a novel approach named as Continual Learning with Efficient Architecture Search, or CLEAS in short. CLEAS works closely with neural architecture search (NAS) which leverages reinforcement learning techniques to search for the best neural architecture that fits a new task. In particular, we design a neuron-level NAS controller that decides which old neurons from previous tasks should be reused (knowledge transfer), and which new neurons should be added (to learn new knowledge). Such a fine-grained controller allows finding a very concise architecture that can fit each new task well. Meanwhile, since we do not alter the weights of the reused neurons, we perfectly memorize the knowledge learned from previous tasks. We evaluate CLEAS on numerous sequential classification tasks, and the results demonstrate that CLEAS outperforms other state-ofthe-art alternative methods, achieving higher classification accuracy while using simpler neural architectures.

show abstract

Section: Related Workmentioning

confidence: 99%

Efficient Architecture Search for Continual Learning

Gao¹,

Zhipeng²,

Klabjan³

2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Once the supernet is trained, each sampled structure can be directly and quickly evaluated without training from scratch. In our work, we also leverage the weight sharing strategy and build a one-shot model [8] with TextNAS [22] search space, while random search is adopted to sample architectures. Besides, we incorporate knowledge distillation and efficiency constraints as search hints to obtain effective and light student models.…”

Section: Related Workmentioning

confidence: 99%

“…Neural Architecture Search: The task is to find an effective and efficient architecture for AutoADR sub-model through teacherstudent framework. We adopt the search space in TextNAS [22] and build a corresponding network subsuming all possible architectures, which is called supernet [8] for the one-shot search algorithm. To apply knowledge distillation, we train the supernet with uniform sampling under the guidance of the soft predictions from teacher model.…”

Section: Autoadr 41 Overviewmentioning

confidence: 99%

“…EfficientNet [19] proposed a simple and highly effective compound scaling method that enables easy scaling up of a backbone model and beats the performances of human-designed image recognition architectures by a large margin. In the NLP domain, some of us proposed TextNAS [22], which designed a novel search space tailored for text representation and achieved state-of-the-art performances on various natural language processing tasks when pre-training was not applied.…”

Section: Introductionmentioning

confidence: 99%

“…The NAS procedure is performed jointly with knowledge distillation in an iterative manner, so that NAS and knowledge distillation will collaborate with each other to find a tailored model architecture to achieve our goals on both accuracy and efficiency. Moreover, following TextNAS [22], we leverage a customized search space for text representation, which consists of a multi-path mixture of convolutional, recurrent, pooling, and self-attention layers. In this way, we can explore the best composition of different layers for the Ad Relevance prediction problem.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

AutoADR

Chen

Yang

Sun

et al. 2020

Proceedings of the 29th ACM International Conference on Information &Amp; Knowledge Management

Self Cite

View full text Add to dashboard Cite

Large-scale pre-trained models have attracted extensive attention in the research community and shown promising results on various tasks of natural language processing. However, these pre-trained models are memory and computation intensive, hindering their deployment into industrial online systems like Ad Relevance. Meanwhile, how to design an effective yet efficient model architecture is another challenging problem in online Ad Relevance. Recently, AutoML shed new lights on architecture design, but how to integrate it with pre-trained language models remains unsettled. In this paper, we propose AutoADR (Automatic model design for AD Relevance)-a novel end-to-end framework to address this challenge, and share our experience to ship these cutting-edge techniques into online Ad Relevance system at Microsoft Bing. Specifically, AutoADR leverages a one-shot neural architecture search algorithm to find a tailored network architecture for Ad Relevance. The search process is simultaneously guided by knowledge distillation from a large pre-trained teacher model (e.g. BERT), while taking the online serving constraints (e.g. memory and latency) into consideration. We add the model designed by AutoADR as a sub-model into the production Ad Relevance model. This additional sub-model improves the Precision-Recall AUC (PR AUC) on top of the original Ad Relevance model by 2.65X of the normalized shipping bar. More importantly, adding this automatically designed sub-model leads to a statistically significant 4.6% Bad-Ad ratio reduction in online A/B testing. This model has been shipped into Microsoft Bing Ad Relevance Production model.

show abstract

TextNAS: A Neural Architecture Search Space tailored for Text Representation

Cited by 2 publications

References 15 publications

Efficient Architecture Search for Continual Learning

Efficient Architecture Search for Continual Learning

AutoADR

Contact Info

Product

Resources

About