Adversarial Training for High-Stakes Reliability

Abstract

In the future, powerful AI systems may be deployed in high-stakes settings, where a single failure could be catastrophic. One technique for improving AI safety in high-stakes settings is adversarial training, which uses an adversary to generate examples to train on in order to achieve better worst-case performance. In this work, we used a language generation task as a testbed for achieving high reliability through adversarial training. We created a series of adversarial training techniques-including a tool that assists human adversaries-to find and eliminate failures in a classifier that filters text completions suggested by a generator. In our simple "avoid injuries" task, we determined that we can set very conservative classifier thresholds without significantly impacting the quality of the filtered outputs. With our chosen thresholds, filtering with our baseline classifier decreases the rate of unsafe completions from about 2.4% to 0.003% on in-distribution data, which is near the limit of our ability to measure. We found that adversarial training significantly increased robustness to the adversarial attacks that we trained on, without affecting in-distribution performance. We hope to see further work in the high-stakes reliability setting, including more powerful tools for enhancing human adversaries and better ways to measure high levels of reliability, until we can confidently rule out the possibility of catastrophic deployment-time failures of powerful models.

A Holistic Approach to Undesired Content Detection in the Real World

“…Mishkin et al (2022) describes an operational process for doing red-teaming with external experts. Ziegler et al (2022) designed a tool to efficiently assist human adversaries to identify failures in a classifier. Models trained with red-teaming are found to be more robust to adversarial attack (Dinan et al 2019;Ziegler et al 2022) and humanin-the-loop dynamic data collection can efficiently improve model performance Kiela et al 2021).…”

Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned

“…Some of these efforts augment humans (through guidelines, templates, programmatic generation of attacks, and various combinations thereof) to devise test cases that cause systems to fail [45,46,29,21,30,55,6,23]. Others use humans in the loop to continuously and dynamically build, break, and fix [20] models in order to continuously make them more robust to failure modes [40,32,55,61]. Finally, a large body of work aims to learn adversarial examples that cause downstream models to produce spurious outputs [50], some of which are reviewed in [59].…”