Efficient Training of Language Models to Fill in the Middle

Abstract

We show that autoregressive language models can learn to infill text after we apply a straightforward transformation to the dataset, which simply moves a span of text from the middle of a document to its end. While this data augmentation has garnered much interest in recent years, we provide extensive evidence that training models with a large fraction of data transformed in this way does not harm the original left-to-right generative capability, as measured by perplexity and sampling evaluations across a wide range of scales. Given the usefulness, simplicity, and efficiency of training models to fill-in-the-middle (FIM), we suggest that future autoregressive language models be trained with FIM by default. To this end, we run a series of ablations on key hyperparameters, such as the data transformation frequency, the structure of the transformation, and the method of selecting the infill span. We use these ablations to prescribe strong default settings and best practices to train FIM models. We have released our best infilling model trained with best practices in our API, and release our infilling benchmarks to aid future research.

Assessment of chemistry knowledge in large language models that generate code

“…There is a strong correlation between the model parameter count and accuracy, 25 so we focus only on the largest models with more than 1B parameters. The architectures of models are all decoder-only like with the ability to insert completions, 26 (except when noted). The rst model is a GPT-3 12B ne-tuned on code (Codex) abbreviated as "cushman".…”

Assessment of chemistry knowledge in large language models that generate code

“…There is a strong correlation between model parameter count and accuracy, 24 so we focus only on the largest models with more than 1B parameters. The architectures of models are all decoder-only like GPT-3 3 with the ability to insert completions, 25 (except when noted). The first model is a GPT-3 12B fine-tuned on code (Codex) abbreviated as "cushman."…”

Deep Conditional Measure Quantization

“…Other approaches to using DNNs for interpolation involve using the DNN to learn a probabilistic model of the data [5], and generate the interpolated values using the learned data distribution (with applications to natural language processing and time series analysis). In NLP the possibility of language models to learn to infill (missing parts of ) text [2] can also be considered close to a extrapolation method.…”