Learning to summarize from human feedback

Abstract

As language models become more powerful, training and evaluation are increasingly bottlenecked by the data and metrics used for a particular task. For example, summarization models are often trained to predict human reference summaries and evaluated using ROUGE, but both of these metrics are rough proxies for what we really care about-summary quality. In this work, we show that it is possible to significantly improve summary quality by training a model to optimize for human preferences. We collect a large, high-quality dataset of human comparisons between summaries, train a model to predict the human-preferred summary, and use that model as a reward function to fine-tune a summarization policy using reinforcement learning. We apply our method to a version of the TL;DR dataset of Reddit posts [57] and find that our models significantly outperform both human reference summaries and much larger models fine-tuned with supervised learning alone. Our models also transfer to CNN/DM news articles [21], producing summaries nearly as good as the human reference without any news-specific fine-tuning. 2 We conduct extensive analyses to understand our human feedback dataset and fine-tuned models. 3 We establish that our reward model generalizes to new datasets, and that optimizing our reward model results in better summaries than optimizing ROUGE according to humans. We hope the evidence from our paper motivates machine learning researchers to pay closer attention to how their training loss affects the model behavior they actually want. * This was a joint project of the OpenAI Reflection team. Author order was randomized amongst {LO, JW, DZ, NS}; CV and RL were full-time contributors for most of the duration. PC is the team lead.2 Samples from all of our models can be viewed on our website. 3 We provide inference code for our 1.3B models and baselines, as well as a model card, here. We are not able to release our dataset of human preferences at this time.Preprint. Under review.

WebGPT: Browser-assisted question-answering with human feedback

“…We therefore collected examples of humans using the browser to answer questions, which we call demonstrations. However, training on demonstrations alone does not directly optimize answer quality, and is unlikely to lead far beyond human performance [Stiennon et al, 2020]. We therefore collected pairs of model-generated answers to the same question, and asked humans which one they preferred, which we call comparisons.…”

“…Starting from the BC model with the final unembedding layer removed, we trained a model to take in a question and an answer with references, and output a scalar reward. Following Stiennon et al [2020], the reward represents an Elo score, scaled such that the difference between two scores represents the logit of the probability that one will be preferred to the other by the human labelers. The reward model is trained using a cross-entropy loss, with the comparisons as labels.…”

“…Instead of trying to improve these ingredients, we focus on combining them using more faithful training objectives. Following Stiennon et al [2020], we use human feedback to directly optimize answer quality, allowing us to achieve performance competitive with humans.…”

A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Models

“…The generated texts are regarded as augmentation data to help improve the classification performance. Moreover, Stiennon et al [2020] use a RL-based approach on the task of English summarization, which fine-tunes the PLM by combining with human feedbacks.…”

Integrating Human-in-the-loop into Swarm Learning for Decentralized Fake News Detection

“…Human-in-the-loop can be applied to improve the performance of machine learning models by integrating human knowledge and experience for data analytics [24]. For example, human can significantly reduce algorithm bias in the training and inference in terms of human feedback for various tasks in the field of natural language processing (NLP) such as text classification [27], syntactic and semantic parsing [28], topic modeling [29], text summarization [30], and sentiment analysis [31]. The general framework is shown in Figure 3.…”