Future User Engagement Prediction and Its Application to Improve the Sensitivity of Online Experiments

Drutsa, Alexey; Gusev, Gleb; Serdyukov, Pavel

doi:10.1145/2736277.2741116

Cited by 38 publications

(18 citation statements)

References 27 publications

(120 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Online controlled experiments, such as A/B testing or interleaving, have become widely used techniques for controlling and improving search quality based on data-driven decisions [19]. This methodology has been adopted widely [3,7,9,31]. An A/B test is a between-subject test designed to compare two variants of a method (e.g., ranking on the SERP, ad ranking, colors and fonts of the web result title) at the same time by exposing them to two user groups and by measuring the difference between them in terms of a key metric (e.g., revenue, number of visits, etc.…”

Section: Background and Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Towards Learning Reward Functions from User Interactions

Kiseleva

Rijke

et al. 2017

Proceedings of the ACM SIGIR International Conference on Theory of Information Retrieval

View full text Add to dashboard Cite

In the physical world, people have dynamic preferences, e.g., the same situation can lead to satisfaction for some humans and to frustration for others. Personalization is called for. The same observation holds for online behavior with interactive systems. It is natural to represent the behavior of users who are engaging with interactive systems such as a search engine or a recommender system, as a sequence of actions where each next action depends on the current situation and the user reward of taking a particular action. By and large, current online evaluation metrics for interactive systems such as search engines or recommender systems, are static and do not reflect differences in user behavior. They rarely capture or model the reward experienced by a user while interacting with an interactive system. We argue that knowing a user's reward function is essential for an interactive system as both for learning and evaluation. We propose to learn users' reward functions directly from observed interaction traces. In particular, we present how users' reward functions can be uncovered directly using inverse reinforcement learning techniques. We also show how to incorporate user features into the learning process. Our main contribution is a novel and dynamic approach to restore a user's reward function. We present an analytic approach to this problem and complement it with initial experiments using the interaction logs of a cultural heritage institution that demonstrate the feasibility of the approach by uncovering different reward functions for different user groups. CCS CONCEPTS• Information systems → Users and interactive retrieval; • Computing methodologies → Inverse reinforcement learning; Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org.

show abstract

Section: Background and Related Workmentioning

confidence: 99%

“…There are many existing studies towards better online evaluation that are devoted to improving the sensitivity of our measurement methods [28], inventing new metrics [8,10] or improving existing ones [9]. An important goal of recent studies is to make metrics more consistent with long-term goals [19].…”

Section: Background and Related Workmentioning

confidence: 99%

Towards Learning Reward Functions from User Interactions

Kiseleva

Rijke

et al. 2017

Proceedings of the ACM SIGIR International Conference on Theory of Information Retrieval

View full text Add to dashboard Cite

show abstract

“…While loyalty and popularity essentially make sense for relative comparison of websites, activity enables measuring engagement for a particular website independently of other websites. The most commonly used activity metrics include number of queries per session, number of clicks, number of clicks per query, dwell (presence) time (see e.g., [10,32]). …”

Section: Related Workmentioning

confidence: 99%

Benchmarking Exploratory OLAP

Djedaini

Furtado

Labroche

et al. 2017

Performance Evaluation and Benchmarking. Traditional - Big Data - Interest of Things

View full text Add to dashboard Cite

Abstract. Supporting interactive database exploration (IDE) is a problem that attracts lots of attention these days. Exploratory OLAP (OnLine Analytical Processing) is an important use case where tools support navigation and analysis of the most interesting data, using the best possible perspectives. While many approaches were proposed (like query recommendation, reuse, steering, personalization or unexpected data recommendation), a recurrent problem is how to assess the effectiveness of an exploratory OLAP approach. In this paper we propose a benchmark framework to do so, that relies on an extensible set of user-centric metrics that relate to the main dimensions of exploratory analysis. Namely, we describe how to model and simulate user activity, how to formalize our metrics and how to build exploratory tasks to properly evaluate an IDE system under test (SUT). To the best of our knowledge, this is the first proposal of such a benchmark. Experiments are two-fold: first we evaluate the benchmark protocol and metrics based on synthetic SUTs whose behavior is well known. Second, we concentrate on two different recent SUTs from IDE literature that are evaluated and compared with our benchmark. Finally, potential extensions to produce an industrystrength benchmark are listed in the conclusion.

show abstract

“…Recently online controlled experiments, such as A/B testing, have become widely used a technique for controlling and improving search quality based on data-driven decisions [155]. This methodology has been adopted by many leading search companies such as Bing [62], Google [223], Facebook [20], and Yandex [71]. An A/B test is designed to compare two variants of a method (e. g. ranking on SERP, ads ranking at the same time by exposing them to two user groups and by measuring the difference between them in terms of a key metric (e. g. the revenue, the number of visits, etc.…”

Section: Search Quality Evaluationmentioning

confidence: 99%

“…), also known as an overall evaluation criterion. There are many existing studies towards better online evaluation which were devoted to inventing new metrics [70,73] or improving existing ones [71]. The main goal of these studies was to make these metrics more consistent with the long-term goals [155].…”

Section: Search Quality Evaluationmentioning

confidence: 99%

Using Contextual Information to Understand Searching and Browsing Behavior

Kiseleva

2015

Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

Future User Engagement Prediction and Its Application to Improve the Sensitivity of Online Experiments

Cited by 38 publications

References 27 publications

Towards Learning Reward Functions from User Interactions

Towards Learning Reward Functions from User Interactions

Benchmarking Exploratory OLAP

Using Contextual Information to Understand Searching and Browsing Behavior

Contact Info

Product

Resources

About