Predictability and Surprise in Large Generative Models

Ganguli, Deep; Hernandez, Danny; Lovitt, Liane; Askell, Amanda; Bai, Yuntao; Chen, Anna; Conerly, Tom; Dassarma, Nova; Drain, Dawn; Elhage, Nelson; Showk, Sheer El; Fort, Stanislav; Hatfield-Dodds, Zac; Henighan, Tom; Johnston, Scott G; Jones, Andy; Joseph, Nicholas; Kernian, Jackson; Kravec, Shauna; Mann, Ben; Nanda, Neel; Kamal, Ndousse,; Olsson, Catherine; Amodei, Daniela; Brown, Tom; Kaplan, Jared; McCandlish, Sam; Olah, Christopher; Amodei, Dario; Clark, Jack

doi:10.1145/3531146.3533229

Cited by 113 publications

(80 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Leaderboards that integrate conventional benchmarking with analysis can be especially helpful by making this largely automatic (Wang et al, 2018;Dua et al, 2019;Gehrmann et al, 2021;. More broadly, careful analysis work, targeted at broadly understanding the capacities of capable models, will be valuable in helping to forecast and mitigate the worst risks from future systems (Elhage et al, 2021;Ganguli et al, 2022).…”

Section: Ways To Do Bettermentioning

confidence: 99%

The Dangers of Underclaiming: Reasons for Caution When Reporting How NLP Systems Fail

Bowman¹

2022

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

View full text Add to dashboard Cite

Researchers in NLP often frame and discuss research results in ways that serve to deemphasize the field's successes, often in response to the field's widespread hype. Though wellmeaning, this has yielded many misleading or false claims about the limits of our best technology. This is a problem, and it may be more serious than it looks: It harms our credibility in ways that can make it harder to mitigate present-day harms, like those involving biased systems for content moderation or resume screening. It also limits our ability to prepare for the potentially enormous impacts of more distant future advances. This paper urges researchers to be careful about these claims and suggests some research directions and communication strategies that will make it easier to avoid or rebut them.

show abstract

Section: Ways To Do Bettermentioning

confidence: 99%

The Dangers of Underclaiming: Reasons for Caution When Reporting How NLP Systems Fail

Bowman¹

2022

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

View full text Add to dashboard Cite

show abstract

“…Novel AI capabilities continue to emerge, 76 increasing the urgency to align AI with humans. Legal standards can serve as a pillar of AI goal specification practices.…”

Section: Discussionmentioning

confidence: 99%

Large Language Models as Fiduciaries: A Case Study Toward Robustly Communicating With Artificial Intelligence Through Legal Standards

Nay

2023

SSRN Journal

View full text Add to dashboard Cite

“…Value lock-in could occur when our technology perpetuates the values of a particular powerful group, or it could occur when groups get stuck in a poor equilibrium that is robust to attempts to get unstuck. Emergent functionality could be hazardous because models demonstrate unexpected, qualitatively different behavior as they become more competent [26,57], so a loss of control becomes more likely when new capabilities or goals spontaneously emerge. Deception is commonly incentivized, and smarter agents are more capable of succeeding at deception; we can be less sure of our models if we fail to find a way to make them assert only what they hold to be true.…”

Section: Speculative Hazards and Failure Modesmentioning

confidence: 99%

“…6. Emergent functionality: Capabilities and novel functionality can spontaneously emerge in today's AI systems [26,57], even though these capabilities were not anticipated by system designers. If we do not know what capabilities systems possess, systems become harder to control or safely deploy.…”

mentioning

confidence: 99%

X-Risk Analysis for AI Research

Hendrycks¹,

Mazeika²

2022

Preprint

View full text Add to dashboard Cite

Artificial intelligence (AI) has the potential to greatly improve society, but as with any powerful technology, it comes with heightened risks and responsibilities. Current AI research lacks a systematic discussion of how to manage long-tail risks from AI systems, including speculative long-term risks. Keeping in mind the potential benefits of AI, there is some concern that building ever more intelligent and powerful AI systems could eventually result in systems that are more powerful than us; some say this is like playing with fire and speculate that this could create existential risks (x-risks). To add precision and ground these discussions, we provide a guide for how to analyze AI x-risk, which consists of three parts: First, we review how systems can be made safer today, drawing on time-tested concepts from hazard analysis and systems safety that have been designed to steer large processes in safer directions. Next, we discuss strategies for having long-term impacts on the safety of future systems. Finally, we discuss a crucial concept in making AI systems safer by improving the balance between safety and general capabilities. We hope this document and the presented concepts and tools serve as a useful guide for understanding how to analyze AI x-risk.

show abstract

Predictability and Surprise in Large Generative Models

Cited by 113 publications

References 10 publications

The Dangers of Underclaiming: Reasons for Caution When Reporting How NLP Systems Fail

The Dangers of Underclaiming: Reasons for Caution When Reporting How NLP Systems Fail

Large Language Models as Fiduciaries: A Case Study Toward Robustly Communicating With Artificial Intelligence Through Legal Standards

X-Risk Analysis for AI Research

Contact Info

Product

Resources

About