Off-policy Learning in Two-stage Recommender Systems

Ma, Jiaqi; Zhao, Zhe; Yi, Xinyang; Yang, Ji Hoon; Chen, Minmin; Tang, Jiaxi; Hong, Lichan; H., Ed

doi:10.1145/3366423.3380130

Cited by 66 publications

(47 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Ma et al [64] extends the policy correction gradient estimator into a two-stage setting which are 𝑝 (𝑠 𝑡 , 𝑎 𝑝 ) and 𝑞(𝑠 𝑡 , 𝑎|𝑎 𝑝 ) and the policy can be written as ∑︁…”

Section: Model-free Deep Reinforcement Learning Based Methodsmentioning

confidence: 99%

A Survey of Deep Reinforcement Learning in Recommender Systems: A Systematic Review and Future Directions

Chen¹,

Yao²,

McAuley³

et al. 2021

Preprint

View full text Add to dashboard Cite

In light of the emergence of deep reinforcement learning (DRL) in recommender systems research and several fruitful results in recent years, this survey aims to provide a timely and comprehensive overview of the recent trends of deep reinforcement learning in recommender systems. We start with the motivation of applying DRL in recommender systems. Then, we provide a taxonomy of current DRL-based recommender systems and a summary of existing methods. We discuss emerging topics and open issues, and provide our perspective on advancing the domain. This survey serves as introductory material for readers from academia and industry into the topic and identifies notable opportunities for further research.

show abstract

Section: Model-free Deep Reinforcement Learning Based Methodsmentioning

confidence: 99%

A Survey of Deep Reinforcement Learning in Recommender Systems: A Systematic Review and Future Directions

Chen¹,

Yao²,

McAuley³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Practical applications, however, are still limited, especially for offline RL (such as [5,25,27,31,43,47]). We attribute this to the lack of application-specific simulation environments that provide useful insights for specific research questions.…”

Section: Towards Practical Research Of Offline Rl In Recsys and Rtbmentioning

confidence: 99%

“…We attribute this to the lack of application-specific simulation environments that provide useful insights for specific research questions. For example, RecSys/RTB are unique regarding their huge action space and highly stochastic and delayed rewards [4,25,55]. Therefore, we need to build a simulation platform imitating such specific characteristics to better understand the empirical performance of offline RL/OPE methods in these particular situations.…”

Section: Towards Practical Research Of Offline Rl In Recsys and Rtbmentioning

confidence: 99%

See 1 more Smart Citation

Accelerating Offline Reinforcement Learning Application in Real-Time Bidding and Recommendation: Potential Use of Simulation

Kiyohara¹,

K²,

Saito³

2021

Preprint

View full text Add to dashboard Cite

In recommender systems (RecSys) and real-time bidding (RTB) for online advertisements, we often try to optimize sequential decision making using bandit and reinforcement learning (RL) techniques. In these applications, offline reinforcement learning (offline RL) and off-policy evaluation (OPE) are beneficial because they enable safe policy optimization using only logged data without any risky online interaction. In this position paper, we explore the potential of using simulation to accelerate practical research of offline RL and OPE, particularly in RecSys and RTB. Specifically, we discuss how simulation can help us conduct empirical research of offline RL and OPE.We take a position to argue that we should effectively use simulations in the empirical research of offline RL and OPE. To refute the counterclaim that experiments using only real-world data are preferable, we first point out the underlying risks and reproducibility issue in real-world experiments. Then, we describe how these issues can be addressed by using simulations. Moreover, we show how to incorporate the benefits of both real-world and simulation-based experiments to defend our position. Finally, we also present an open challenge to further facilitate practical research of offline RL and OPE in RecSys and RTB, with respect to public simulation platforms. As a possible solution for the issue, we show our ongoing open source project and its potential use case. We believe that building and utilizing simulation-based evaluation platforms for offline RL and OPE will be of great interest and relevance for the RecSys and RTB community.

show abstract

“…To solve this issue, we introduce gradient estimation based on Monte-Carlo sampling. Our approach is similar to that of Ma et al [15], however, we are estimating gradients of variance instead of general performance.…”

Section: Monte-carlo-based Derivativesmentioning

confidence: 99%

Taking the Counterfactual Online: Efficient and Unbiased Online Evaluation for Ranking

Oosterhuis

Rijke

2020

Proceedings of the 2020 ACM SIGIR on International Conference on Theory of Information Retrieval

View full text Add to dashboard Cite

Counterfactual evaluation can estimate Click-Through-Rate (CTR) differences between ranking systems based on historical interaction data, while mitigating the effect of position bias and item-selection bias. We introduce the novel Logging-Policy Optimization Algorithm (LogOpt), which optimizes the policy for logging data so that the counterfactual estimate has minimal variance. As minimizing variance leads to faster convergence, LogOpt increases the data-efficiency of counterfactual estimation. LogOpt turns the counterfactual approach-which is indifferent to the logging policyinto an online approach, where the algorithm decides what rankings to display. We prove that, as an online evaluation method, LogOpt is unbiased w.r.t. position and item-selection bias, unlike existing interleaving methods. Furthermore, we perform large-scale experiments by simulating comparisons between thousands of rankers. Our results show that while interleaving methods make systematic errors, LogOpt is as efficient as interleaving without being biased.

show abstract

Off-policy Learning in Two-stage Recommender Systems

Cited by 66 publications

References 22 publications

A Survey of Deep Reinforcement Learning in Recommender Systems: A Systematic Review and Future Directions

A Survey of Deep Reinforcement Learning in Recommender Systems: A Systematic Review and Future Directions

Accelerating Offline Reinforcement Learning Application in Real-Time Bidding and Recommendation: Potential Use of Simulation

Taking the Counterfactual Online: Efficient and Unbiased Online Evaluation for Ranking

Contact Info

Product

Resources

About