What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers

Abstract

GPT-3 shows remarkable in-context learning ability of large-scale language models (LMs) trained on hundreds of billion scale data. Here we address some remaining issues less reported by the GPT-3 paper, such as a non-English LM, the performances of different sized models, and the effect of recently introduced prompt optimization on in-context learning. To achieve this, we introduce Hy-perCLOVA, a Korean variant of 82B GPT-3 trained on a Korean-centric corpus of 560B tokens. Enhanced by our Korean-specific tokenization, HyperCLOVA with our training configuration shows state-of-the-art in-context zero-shot and few-shot learning performances on various downstream tasks in Korean. Also, we show the performance benefits of promptbased learning and demonstrate how it can be integrated into the prompt engineering pipeline. Then we discuss the possibility of materializing the No Code AI paradigm by providing AI prototyping capabilities to nonexperts of ML by introducing HyperCLOVA studio, an interactive prompt engineering interface. Lastly, we demonstrate the potential of our methods with three successful in-house applications.

nuQmm: Quantized MatMul for Efficient Inference of Large-Scale Generative Language Models

“…Pre-trained extreme-scale language models (e.g., GPT-3 (175B) [6], HyperCLOVA (204B) [8], and Megatron Turing NLG (530B) [10]) are usually not publicly available. Thus, in this work, our detailed analysis of group-wise quantization and nuQmm is limited to relatively smaller models (such as GPT Neo).…”

“…In addition to the architectural advantages of the Transformer to scale the model size, generative LMs are increasing the number of parameters (as depicted in Table I) because: 1) self-supervised learning alleviates the burden of the expensive labeling process and 2) scaling-law [6], [7] provides a predictable performance on the cross-entropy loss as the model size increases. Surprising qualitative evaluation results (e.g., human-like writing) of extreme-scale LMs also fueled the competition in model size [8], [9].…”

“…As for NLP tasks, it is reported that LM performance follows predictable power-law scaling as a function of model size [6], [7]. Since then, especially for generative LMs, researchers proposed numerous large-scale models including GPT-3 (175B) [6], HyperCLOVA (204B) [8], Gopher (280B) [9], and Megatron Turing NLG (530B) [10]. Note that models of billions of parameters cannot be accommodated by one GPU since GPU memory size is sacrificed and limited in order to enhance memory bandwidth [11], [12].…”

On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model

“…various large-scale in-context language models have been proposed (Black et al, 2021;Kim et al, 2021;Zeng et al, 2021;Rae et al, 2021;Hoffmann et al, 2022;Chowdhery et al, 2022).…”

“…In addition, analysis on the relation between the validation perplexity of a language model and incontext learning performance is still less investigated. Previous research on in-context learning implicitly assumes that perplexity is predictive of incontext learning performance by showing scaling law property of their model (Kaplan et al, 2020;Brown et al, 2020;Kim et al, 2021). Rae et al (2021) also use perplexity for the hyperparameter selection on corpus reweighting in the pretraining of their in-context learner.…”