On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model

Abstract

Many recent studies on large-scale language models have reported successful in-context zero-and few-shot learning ability. However, the in-depth analysis of when in-context learning occurs is still lacking. For example, it is unknown how in-context learning performance changes as the training corpus varies. Here, we investigate the effects of the source and size of the pretraining corpus on in-context learning in HyperCLOVA, a Korean-centric GPT-3 model. From our in-depth investigation, we introduce the following observations: (1) incontext learning performance heavily depends on the corpus domain source, and the size of the pretraining corpus does not necessarily determine the emergence of in-context learning, (2) in-context learning ability can emerge when a language model is trained on a combination of multiple corpora, even when each corpus does not result in in-context learning on its own, (3) pretraining with a corpus related to a downstream task does not always guarantee the competitive in-context learning performance of the downstream task, especially in the few-shot setting, and (4) the relationship between language modeling (measured in perplexity) and in-context learning does not always correlate: e.g., low perplexity does not always imply high in-context few-shot learning performance.

A Survey on In-context Learning

“…Pre-training Stage We first introduce influence factors in the LLM pretraining stage. Shin et al (2022a) investigated the influence of the pretraining corpora. They found that the domain source is more important than the corpus size.…”

“…Pretraining corpus domain (Shin et al, 2022a) Pretraining corpus combination (Shin et al, 2022a) Number of model parameters (Wei et al, 2022b;Brown et al, 2020) Number of pretraining steps (Wei et al, 2022b) Inference Label space exposure (Min et al, 2022c) Demonstration input distribution (Min et al, 2022c) Format of input-label pairing (Min et al, 2022c) Demonstration input-label mapping (Min et al, 2022c; Demonstration sample ordering (Lu et al, 2022) Demonstration-query similarity Table 3: Summary of factors that have a relatively strong correlation to ICL performance. ios.…”

“…As investigated by Shin et al (2022b), language model objectives are not equal to ICL abilities. Researchers have proposed to bridge the gap between pretraining objectives and ICL through intermediate tuning before inference (Section 4), which shows promising performance improvements.…”

Distinguishability Calibration to In-Context Learning

“…However, as in-context learning does not require updating PLM parameters, there arises the problem of distribution mismatch between the data used for LM pre-training and the test samples used in in-context learning, which hinders the full exploitation of the knowledge encoded in PLMs Ge et al, 2022;Shin et al, 2022). To alleviate the context shift, existing methods rely on prior knowledge to increase the overlapping between the two distributions.…”

School Mathematics Textbooks in China

“…Large Language Models (LLMs) have succeeded in advancing the state-of-the-arts for many Natural Language Processing (NLP) tasks [Devlin et al, 2019, Brown et al, 2020, Rae et al, 2021, Thoppilan et al, 2022, Chowdhery et al, 2022, Scao et al, 2022, Zhang et al, 2022b, Bai et al, 2022, Touvron et al, 2023, benefiting from the ultra-large-scale training corpora and computation resources. To unleash the LLMs' power of adaptation on unseen tasks without any parameter updates, in-context learning (ICL) has become one of the flourishing research topics, aiming at generating the prediction by conditioning on a few labeled exemplars (Figure 1 (a)) [Dong et al, 2023, Zhao et al, 2021, Shin et al, 2022, Lu et al, 2022.…”