Towards Learning Reward Functions from User Interactions

Abstract

In the physical world, people have dynamic preferences, e.g., the same situation can lead to satisfaction for some humans and to frustration for others. Personalization is called for. The same observation holds for online behavior with interactive systems. It is natural to represent the behavior of users who are engaging with interactive systems such as a search engine or a recommender system, as a sequence of actions where each next action depends on the current situation and the user reward of taking a particular action. By and large, current online evaluation metrics for interactive systems such as search engines or recommender systems, are static and do not reflect differences in user behavior. They rarely capture or model the reward experienced by a user while interacting with an interactive system. We argue that knowing a user's reward function is essential for an interactive system as both for learning and evaluation. We propose to learn users' reward functions directly from observed interaction traces. In particular, we present how users' reward functions can be uncovered directly using inverse reinforcement learning techniques. We also show how to incorporate user features into the learning process. Our main contribution is a novel and dynamic approach to restore a user's reward function. We present an analytic approach to this problem and complement it with initial experiments using the interaction logs of a cultural heritage institution that demonstrate the feasibility of the approach by uncovering different reward functions for different user groups. CCS CONCEPTS• Information systems → Users and interactive retrieval; • Computing methodologies → Inverse reinforcement learning; Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org.

Systematic review of training methods for conversational systems: the potential of datasets validated with user experience

“…Less frequently mentioned methods include the use of interaction logs (Z. Li, Kiseleva, De Rijke, & Grotov, 2017) and autonomous data collection with network-shared knowledge (Ahrndt, Lützenberger, & Prochnow, 2016).…”

Proceedings of the 10th ACM Conference on Recommender Systems

“…All in all, it can be particularly challenging to translate highlevel complex RS goals to an individual numeric reward signal [9,11], and there is much work to be explored in the field of reward engineering for RS. Possible topics are: designing/modeling goals [8,15], reward hacking [2,16,23], incorporating artificial curiosity/intrinsic motivation [4,17,20,21] (e.g., by explicitly rewarding the RS for learning new information about users) and inverse RL [14].…”

Optimizing Interactive Systems with Data-Driven Objectives

“…Currently, optimizing interactive systems relies on explicit assumptions about users' objectives in terms of their needs and frustrations (Li et al, 2017b). Commonly, an objective function is manually designed for a particular task to reflect the quality of an interactive system, e.g., in terms of user satisfaction (Kelly, 2009(Kelly, , 2015, user effort (Yilmaz et al, 2014) or other domain-specific metrics, such as relevance judgements in information retrieval (Järvelin and Kekäläinen, 2002;Saracevic, 1975;Saracevic et al, 1988;Drutsa et al, 2015;Dupret and Lalmas, 2013), user feedbacks (e.g.…”