Language Models (Mostly) Know What They Know

Abstract

We study whether language models can evaluate the validity of their own claims and predict which questions they will be able to answer correctly. We first show that larger models are well-calibrated on diverse multiple choice and true/false questions when they are provided in the right format. Thus we can approach self-evaluation on open-ended sampling tasks by asking models to first propose answers, and then to evaluate the probability "P(True)" that their answers are correct. We find encouraging performance, calibration, and scaling for P(True) on a diverse array of tasks. Performance at self-evaluation further improves when we allow models to consider many of their own samples before predicting the validity of one specific possibility. Next, we investigate whether models can be trained to predict "P(IK)", the probability that "I know" the answer to a question, without reference to any particular proposed answer. Models perform well at predicting P(IK) and partially generalize across tasks, though they struggle with calibration of P(IK) on new tasks. The predicted P(IK) probabilities also increase appropriately in the presence of relevant source materials in the context, and in the presence of hints towards the solution of mathematical word problems. We hope these observations lay the groundwork for training more honest models, and for investigating how honesty generalizes to cases where models are trained on objectives other than the imitation of human writing.

Measuring Progress on Scalable Oversight for Large Language Models

“…On QuALITY, unaided humans do poorly, with human-model teams doing substantially better. These results are consistent with Kadavath et al (2022)'s report that our plain pretrained language models tend to be very well calibrated on multiple-choice question answering, but that their calibration degrades after RLHF training for helpfulness, which we use for all runs in this paper. We expect that explicit calibration training on the task would have improved these results for human participants and human-model teams (Lichtenstein and Fischhoff, 1980), which could potentially improve raw accuracy as well by better weighting votes across participants.…”

A Closer Look at the Calibration of Differentially Private Learners

“…Calibrated probability estimates match the true empirical frequencies of an outcome, and calibration is often used to evaluate the quality of uncertainty estimates provided by ML models. Recent works have observed that highly-accurate models that leverage pre-training are often well-calibrated [21,22,23]. However, we find that even pre-trained models are poorly calibrated when they are fine-tuned using DP-SGD.…”

Defending ChatGPT against Jailbreak Attack via Self-Reminder

“…In this work, we aim to apply this psychological self-improvement strategy for human behavior to the behavior of LLMs. Second, the emerging abilities of LLMs to perform self-validation and self-correction, as demonstrated in recent studies [30][31][32] , suggest the possibility of addressing this challenging problem using ChatGPT itself. Third, we draw inspiration from existing Jailbreaks, many of which bypass ChatGPT's moral alignment by guiding it into certain uncontrollable "modes" that will then generate harmful responses.…”

“…Recent studies have been exploring the capacity of large language models to validate and correct their own claims [30][31][32] . For instance, the prior work 31 investigates the ability of language models to evaluate the validity of their claims and predict their ability to answer questions, while the recent study 30 demonstrates the capacity of LLMs for moral correction.…”