Neural Architecture Search for Sentence Classification with BERT

Abstract

Pre training of language models on large text corpora is common practice in Natural Language Processing. Following, fine tuning of these models is performed to achieve the best results on a variety of tasks. In this paper we question the common practice of only adding a single output layer as a classification head on top of the network. We perform an AutoML search to find architectures that outperform the current single layer at only a small compute cost. We validate our classification architecture on a variety of NLP benchmarks from the GLUE dataset. The source code is open-source and free (MIT licensed) software, available at https://github.com/TheMody/NASforSentenceEmbeddingHeads.* We gratefully acknowledge funding by the BMWi (01MK20007E), from the MKW NRW in the project Bias aus KI-Modellen and by the BMBF (01IS19080 A).