The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models

Pan, Alexander; Bhatia, Kush; Steinhardt, Jacob

doi:10.48550/arxiv.2201.03544

Cited by 7 publications

(5 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While some problems, such as reachability and safety, naturally lend themselves to a reward-based formulation, such an interface is often cumbersome and arguably error-prone. This difficulty has been well documented, especially within the RL literature, under different terms including misaligned specification, specification gaming, and reward hacking (Pan, Bhatia, and Steinhardt 2022;Amodei et al 2016;Yuan et al 2019;Skalse et al 2022;Clark and Amodei 2016).…”

Section: Motivationmentioning

confidence: 99%

Omega-Regular Decision Processes

Hahn,

Perez,

Schewe

et al. 2024

AAAI

View full text Add to dashboard Cite

Regular decision processes (RDPs) are a subclass of non-Markovian decision processes where the transition and reward functions are guarded by some regular property of the past (a lookback). While RDPs enable intuitive and succinct representation of non-Markovian decision processes, their expressive power coincides with finite-state Markov decision processes (MDPs). We introduce omega-regular decision processes (ODPs) where the non-Markovian aspect of the transition and reward functions are extended to an omega-regular lookahead over the system evolution. Semantically, these lookaheads can be considered as promises made by the decision maker or the learning agent about her future behavior. In particular, we assume that, if the promised lookaheads are not met, then the payoff to the decision maker is falsum (least desirable payoff), overriding any rewards collected by the decision maker. We enable optimization and learning for ODPs under the discounted-reward objective by reducing them to lexicographic optimization and learning over finite MDPs. We present experimental results demonstrating the effectiveness of the proposed reduction.

show abstract

Section: Motivationmentioning

confidence: 99%

Omega-Regular Decision Processes

Hahn,

Perez,

Schewe

et al. 2024

AAAI

View full text Add to dashboard Cite

show abstract

“…As the agent and environment increase in complexity, it becomes harder to design reward functions that do not reward hack. Recent research provides sub-workflows to prevent this phenomenon, for example via detecting anomalous or aberrant policies (Pan et al, 2022), or via iteratively shaping reward functions (Gajcin et al, 2023) Visualizing the reward landscape, the reward surface over parameters of interest, is an effective tool to understand the complexity and convergence of reward function. The loss function can be complex due to the non-convex and/or NP-hardness (Li et al, 2018), while the reward function can be more complex, which is observed by visualizing the reward landscape (Sullivan et al, 2022).…”

Section: Designing the Reward Functionmentioning

confidence: 99%

A Workflow for building Computationally Rational Models of Human Behavior

Chandramouli,

Shi,

Putkonen

et al. 2024

Preprint

View full text Add to dashboard Cite

Convergent progress in cognitive science and machine learning has given rise to a new approach to modeling human behavior, which can better account for its adaptive nature. This follows the theory of computational rationality, under which behavior adapts to maximize expected utility while constrained by the computational limits implied by their cognition. Building computational models of this theory involves three phases: i) specifying rewards and cognitive resources, ii) formulating a sequential decision-making problem to control such resources to maximize expected reward, and iii) using methods like deep reinforcement learning to approximate the optimal policy solution. However, despite their success in reproducing human-like adaptive behavior across everyday tasks, these models are not easy to build. The design of psychological assumptions and machine learning-decisions concerning policy optimization, parameter inference, and model selection, are all tangled processes rife with pitfalls that hinder the development of valid and effective models. The article, drawing from a decade of work on this approach, lays out a workflow for tackling this challenge and is accompanied with detailed discussion of pros and cons at key decision points.

show abstract

“…This has led to a number of breakthroughs, via capabilities such as in-context learning (Radford et al, 2019;Brown et al, 2020) and chain-of-thought prompting (Wei et al, 2022b). However, it also poses risks: Pan et al (2022) show that scaling up the parameter count of models by as little as 30% can lead to emergent reward hacking.…”

Section: Introductionmentioning

confidence: 99%

Progress measures for grokking via mechanistic interpretability

Nanda¹,

Chan²,

Tom³

et al. 2023

Preprint

View full text Add to dashboard Cite

Neural networks often exhibit emergent behavior, where qualitatively new capabilities arise from scaling up the amount of parameters, training data, or training steps. One approach to understanding emergence is to find continuous progress measures that underlie the seemingly discontinuous qualitative changes. We argue that progress measures can be found via mechanistic interpretability: reverseengineering learned behaviors into their individual components. As a case study, we investigate the recently-discovered phenomenon of "grokking" exhibited by small transformers trained on modular addition tasks. We fully reverse engineer the algorithm learned by these networks, which uses discrete Fourier transforms and trigonometric identities to convert addition to rotation about a circle. We confirm the algorithm by analyzing the activations and weights and by performing ablations in Fourier space. Based on this understanding, we define progress measures that allow us to study the dynamics of training and split training into three continuous phases: memorization, circuit formation, and cleanup. Our results show that grokking, rather than being a sudden shift, arises from the gradual amplification of structured mechanisms encoded in the weights, followed by the later removal of memorizing components.

show abstract

The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models

Cited by 7 publications

References 22 publications

Omega-Regular Decision Processes

Omega-Regular Decision Processes

A Workflow for building Computationally Rational Models of Human Behavior

Progress measures for grokking via mechanistic interpretability

Contact Info

Product

Resources

About