When Do You Need Billions of Words of Pretraining Data?

Abstract

NLP is currently dominated by language models like RoBERTa which are pretrained on billions of words. But what exact knowledge or skills do Transformer LMs learn from large-scale pretraining that they cannot learn from less data? To explore this question, we adopt five styles of evaluation: classifier probing, information-theoretic probing, unsupervised relative acceptability judgments, unsupervised language model knowledge probing, and fine-tuning on NLU tasks. We then draw learning curves that track the growth of these different measures of model ability with respect to pretraining data volume using the MiniBERTas, a group of RoBERTa models pretrained on 1M, 10M, 100M and 1B words. We find that these LMs require only about 10M to 100M words to learn to reliably encode most syntactic and semantic features we test. They need a much larger quantity of data in order to acquire enough commonsense knowledge and other skills required to master typical downstream NLU tasks. The results suggest that, while the ability to encode linguistic features is almost certainly necessary for language understanding, it is likely that other, unidentified, forms of knowledge are the major drivers of recent improvements in language understanding among large pretrained models.

The Rediscovery Hypothesis: Language Models Need to Meet Linguistics

“…Even if part of the probing performance is due to the classifier itself we claim that the difference in the probing performance will be due to the difference in the amount of linguistic knowledge encoded in the representations we manipulate. This conjecture is strengthened by the findings of Zhang et al (2020) who analysed the representations from pretrained miniBERTas 3 and demonstrated that the trends found through edge probing (Tenney et al, 2019b) are the same as those found through better designed probes such as Minimum Description Length (Voita & Titov, 2020). Therefore in our work we adopt edge probing and structural probing for contextualized embeddings.…”

Neural reality of argument structure constructions

“…This finding agrees with Warstadt et al (2020b), who found that larger LMs have an inductive bias towards linguistic generalizations, while smaller LMs have an inductive bias towards surface generalizations; this may explain the success of large LMs on downstream tasks. A small quantity of data (10M tokens) is sufficient for LMs to prefer the constructional sort, indicating that ASCs are relatively easy to learn: roughly on par with other types of linguistic knowledge, and requiring less data than commonsense knowledge (Zhang et al, 2021;Liu et al, 2021).…”

On the Role of Corpus Ordering in Language Modeling

“…Scaling laws (Kaplan et al, 2020) suggest that optimal compute-efficient training involves training very large models on a relatively modest amount of data and indeed, many studies explore the relationship between the volume of training data and consequent model performance (Banko and Brill, 2001;Sun et al, 2017;van Schijndel et al, 2019;Hu et al, 2020;Raffel et al, 2020;Brown et al, 2020;Zhang et al, 2021), generally concluding that rapid improvements in performance are observed as the amount of training data increases, at least until a certain point, after which the improvements slow down.…”