Optimal Policies Tend to Seek Power

Turner, Alexander; Smith, Logan Pearsall; Shah, Rohin; Critch, Andrew; Tadepalli, Prasad

doi:10.48550/arxiv.1912.01683

Cited by 3 publications

(4 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Turner et al [26]'s theorem 33 shows the following result for deterministic dynamics and for single states s = s . We generalize to the stochastic case and to distributions over states.…”

Section: B1 Main Resultsmentioning

confidence: 97%

See 1 more Smart Citation

Avoiding Side Effects in Complex Environments

Turner¹,

Ratzlaff²,

Tadepalli³

2020

Preprint

Self Cite

View full text Add to dashboard Cite

Reward function specification can be difficult, even in simple environments. Realistic environments contain millions of states. Rewarding the agent for making a widget may be easy, but penalizing the multitude of possible negative side effects is hard. In toy environments, Attainable Utility Preservation (AUP) avoids side effects by penalizing shifts in the ability to achieve randomly generated goals. We scale this approach to large, randomly generated environments based on Conway's Game of Life. By preserving optimal value for a single randomly generated reward function, AUP incurs modest overhead, completes the specified task, and avoids side effects. * Equal contribution.Preprint. Under review.

show abstract

“…Turner et al [26]'s theorem 33 shows the following result for deterministic dynamics and for single states s = s . We generalize to the stochastic case and to distributions over states.…”

Section: B1 Main Resultsmentioning

confidence: 97%

“…For example, resources increase average optimal value, while immobility decreases it. Definition (Average optimal value [26]). V * avg (s) := E D V * R (s) .…”

Section: Theoretical Resultsmentioning

confidence: 99%

Avoiding Side Effects in Complex Environments

Turner¹,

Ratzlaff²,

Tadepalli³

2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…In human-AI collaboration, AI alignment is pivotal in ensuring AI systems pursue goals following human values or interests (Bostrom 2014;Russell 2019;Ngo, Chan, and Mindermann 2023). If left unchecked, unintended and undesirable goals, or emergent instrumental goals, such as self-preservation or power-seeking (Turner et al 2023), could have catastrophic consequences, including human extinction (Cotra 2022). Although various research directions and agendas have been proposed, including debate (Irving, Christiano, and Amodei 2018), scalable oversight (Bowman et al 2022), iterated distillation and amplification (Christiano, Shlegeris, and Amodei 2018), and reinforcement learning from human feedback (Christiano et al 2023), the field has not yet converged on an overarching paradigm.…”

Section: Introductionmentioning

confidence: 99%

Taming Simulators: Challenges, Pathways and Vision for the Alignment of Large Language Models

Bereska,

Gavves

2023

AAAI-SS

View full text Add to dashboard Cite

As AI systems continue to advance in power and prevalence, ensuring alignment between humans and AI is crucial to prevent catastrophic outcomes. The greater the capabilities and generality of an AI system, combined with its development of goals and agency, the higher the risks associated with misalignment. While the concept of superhuman artificial general intelligence is still speculative, language models show indications of generality that could extend to generally capable systems. Regarding agency, this paper emphasizes the understanding of prediction-trained models as simulators rather than agents. Nonetheless, agents may emerge accidentally from internal processes, so-called simulacra, or deliberately through fine-tuning with reinforcement learning. As a result, the focus of alignment research shifts towards aligning simulacra, comprehending and mitigating mesa-optimization, and aligning agents derived from prediction-trained models. The paper outlines the challenges of aligning simulators and presents research directions based on this understanding. Additionally, it envisions a future where aligned simulators are critical in fostering successful human-AI collaboration. This vision encompasses exploring emulation approaches and the integration of simulators into cyborg systems to enhance human cognitive abilities. By acknowledging the risks associated with misaligned AI, delving into the concept of simulacra, and presenting strategies for aligning agents and simulacra, this paper contributes to the ongoing efforts to safeguard human values in developing and deploying AI systems.

show abstract

“…Indeed, while capability robustness failures are concerning, objective robustness failures are potentially more dangerous: the only risks from an incapable agent are those of accidents from its incompetence, but the same is not true for an agent that capably pursues an incorrect objective, which can leverage its learned capabilities to visit potentially arbitrarily bad states (as assessed by the actual reward). For example, for most reward functions, the optimal policy will try to avoid being shut down [35]. Our main contributions are:…”

Section: Introductionmentioning

confidence: 99%

Goal Misgeneralization in Deep Reinforcement Learning

Lauro¹,

Koch²,

Lee³

et al. 2021

Preprint

View full text Add to dashboard Cite

We study objective robustness failures, a type of out-of-distribution robustness failure in reinforcement learning (RL). Objective robustness failures occur when an RL agent retains its capabilities off-distribution yet pursues the wrong objective. We provide the first explicit empirical demonstrations of objective robustness failures and argue that this type of failure is critical to address. * Equal contribution Preprint. Under review.

show abstract

Optimal Policies Tend to Seek Power

Cited by 3 publications

References 8 publications

Avoiding Side Effects in Complex Environments

Avoiding Side Effects in Complex Environments

Taming Simulators: Challenges, Pathways and Vision for the Alignment of Large Language Models

Goal Misgeneralization in Deep Reinforcement Learning

Contact Info

Product

Resources

About