Recognising the importance of preference change: A call for a coordinated multidisciplinary research effort in the age of AI

Franklin, Matija; Ashton, Hal; Gorman, Rebecca; Armstrong, Stuart

doi:10.48550/arxiv.2203.10525

Cited by 5 publications

(4 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Another limitation of our formalism is that it can only express preferences for AI influence implicitly, by having influence be optimal under a human's reward function. Allowing such meta-preferences (George, 2001;Franklin et al, 2022) to be expressible explicitly, i.e., allowing reward functions to evaluate transitions between different reward functions, may be useful to more clearly capture notions of the legitimacy of influence and personal autonomy.…”

Section: Limitations and Discussionmentioning

confidence: 99%

“…Preference changes in AI. While there is growing recognition of the importance of accounting for influence (Bezou-Vrakatseli et al, 2023;Hendrycks et al, 2023), manipulation (Carroll et al, 2023), and preference changes (Franklin et al, 2022), there has been limited prior work focusing on operationalizing what should be optimized under preference changes. While some have suggested to aim for preference stationarity (Dean & Morgenstern, 2022), most other prior work which accounts for preference change generally takes either a descriptive stance (Curmei et al, 2022;Hazrati & Ricci, 2022), or an explicit normative stance on what the correct notion of optimality is for their specific setting (Evans et al, 2015;Sanna Passino et al, 2021).…”

Section: Related Workmentioning

confidence: 99%

“…While past work has acknowledged that the static-preference assumption is unrealistic (Franklin et al, 2022), there have only been limited attempts at relaxing it, in part also due to a lack of a clear formal language for grounding notions of alignment which explicitly accounts for preference changes. We introduce a natural extension of Markov Decision Processes: Dynamic Reward MDPs (DR-MDPs), which ac-count for changing preferences by modeling them as changing reward functions.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Estimating and Penalizing Preference Shift in Recommender Systems

Carroll

Hadfield-Menell

Russell

et al. 2021

Fifteenth ACM Conference on Recommender Systems

View full text Add to dashboard Cite

Existing AI alignment approaches assume that preferences are static, which is unrealistic: our preferences change, and may even be influenced by our interactions with AI systems themselves. To clarify the consequences of incorrectly assuming static preferences, we introduce Dynamic Reward Markov Decision Processes (DR-MDPs), which explicitly model preference changes and the AI's influence on them. We show that despite its convenience, the static-preference assumption may undermine the soundness of existing alignment techniques, leading them to implicitly reward AI systems for influencing user preferences in ways users may not truly want. We then explore potential solutions. First, we offer a unifying perspective on how an agent's optimization horizon may partially help reduce undesirable AI influence. Then, we formalize different notions of AI alignment that account for preference change from the outset. Comparing the strengths and limitations of 8 such notions of alignment, we find that they all either err towards causing undesirable AI influence, or are overly risk-averse, suggesting that a straightforward solution to the problems of changing preferences may not exist. As there is no avoiding grappling with changing preferences in real-world settings, this makes it all the more important to handle these issues with care, balancing risks and capabilities. We hope our work can provide conceptual clarity and constitute a first step towards AI alignment practices which explicitly account for (and contend with) the changing and influenceable nature of human preferences.

show abstract

Section: Limitations and Discussionmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Estimating and Penalizing Preference Shift in Recommender Systems

Carroll

Hadfield-Menell

Russell

et al. 2021

Fifteenth ACM Conference on Recommender Systems

View full text Add to dashboard Cite

show abstract

“…In this work, we focus on user's preference shifts, which is a particularly challenging problem that has been gaining more attention lately (Franklin et al, 2022). Any recommender policy will have some influence on user preferences.…”

Section: Introductionmentioning

confidence: 99%

Estimating and Penalizing Induced Preference Shifts in Recommender Systems

Carroll¹,

Hadfield-Menell²,

Russell³

et al. 2022

Preprint

View full text Add to dashboard Cite

The content that a recommender system (RS) shows to users influences them. Therefore, when choosing which recommender to deploy, one is implicitly also choosing to induce specific internal states in users. Even more, systems trained via long-horizon optimization will have direct incentives to manipulate users, e.g. shift their preferences so they are easier to satisfy. In this work we focus on induced preference shifts in users. We argue that -before deployment -system designers should: estimate the shifts a recommender would induce; evaluate whether such shifts would be undesirable; and even actively optimize to avoid problematic shifts. These steps involve two challenging ingredients: estimation requires anticipating how hypothetical policies would influence user preferences if deployed -we do this by using historical user interaction data to train predictive user model which implicitly contains their preference dynamics; evaluation and optimization additionally require metrics to assess whether such influences are manipulative or otherwise unwanted -we use the notion of "safe shifts", that define a trust region within which behavior is safe. In simulated experiments, we show that our learned preference dynamics model is effective in estimating user preferences and how they would respond to new recommenders. Additionally, we show that recommenders that optimize for staying in the trust region can avoid manipulative behaviors while still generating engagement.

show abstract