Jointly Learning to Recommend and Advertise

Zhao, Xiangyu; Zheng, Xudong; Yang, Xiwang; Liu, Xiaobing; Tang, Jiliang

doi:10.1145/3394486.3403384

Cited by 65 publications

(32 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There are some studies targeting recommendation and advertising simultaneously in e-commerce environments [73,123,129]. Pei et al [73] mentions when deploying RS into real-world platforms such as e-commerce scenarios, the expectation is to improve the profit of the system.…”

Section: Model-free Deep Reinforcement Learning Based Methodsmentioning

confidence: 99%

“…A new metric, Gross Merchandise Volume (GMV), is proposed to measure the profitability of the RS to provide a new view about evaluating RS in advertising. Different from GMV, Zhao et al [129] separates recommendation and advertising as two different tasks and proposes the Rec/Ads Mixed display (RAM) framework. RAM designs two agents: a recommendation agent and an advertising agent, where each agent employs a CDQN to conduct the corresponding task.…”

Section: Model-free Deep Reinforcement Learning Based Methodsmentioning

confidence: 99%

See 1 more Smart Citation

A Survey of Deep Reinforcement Learning in Recommender Systems: A Systematic Review and Future Directions

Chen¹,

Yao²,

McAuley³

et al. 2021

Preprint

View full text Add to dashboard Cite

In light of the emergence of deep reinforcement learning (DRL) in recommender systems research and several fruitful results in recent years, this survey aims to provide a timely and comprehensive overview of the recent trends of deep reinforcement learning in recommender systems. We start with the motivation of applying DRL in recommender systems. Then, we provide a taxonomy of current DRL-based recommender systems and a summary of existing methods. We discuss emerging topics and open issues, and provide our perspective on advancing the domain. This survey serves as introductory material for readers from academia and industry into the topic and identifies notable opportunities for further research.

show abstract

Section: Model-free Deep Reinforcement Learning Based Methodsmentioning

confidence: 99%

Section: Model-free Deep Reinforcement Learning Based Methodsmentioning

confidence: 99%

A Survey of Deep Reinforcement Learning in Recommender Systems: A Systematic Review and Future Directions

Chen¹,

Yao²,

McAuley³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…RL-based methods model the ads allocation problem as an MDP and solved it with different RL techniques. Zhao et al (2020) proposes a two-level RL framework to jointly optimize the recommending and advertising strategies. Zhao et al (2021) proposes a DQN architecture to determine the optimal ads and ads position jointly.…”

Section: Related Workmentioning

confidence: 99%

“…Early dynamic slots strategies use some classic algorithms (e.g., Bellman-Ford, unified rank score) to allocate ads slots. Since the feed is presented to the user in a sequence, recent dynamic ads allocation strategies usually model the problem as Markov Decision Process (Sutton, Barto et al 1998) and solve it using reinforcement learning (RL) (Zhang et al 2018;Feng et al 2018;Zhao et al 2020).…”

Section: Introductionmentioning

confidence: 99%

Cross DQN: Cross Deep Q Network for Ads Allocation in Feed

Liao¹,

Wang²,

Wu³

et al. 2021

Preprint

View full text Add to dashboard Cite

E-commerce platforms usually display a mixed list of ads and organic items in feed. One key problem is to allocate the limited slots in the feed to maximize the overall revenue as well as improve user experience, which requires a good model for user preference. Instead of modeling the influence of individual items on user behaviors, the arrangement signal models the influence of the arrangement of items and may lead to a better allocation strategy. However, most of previous strategies fail to model such a signal and therefore result in suboptimal performance. To this end, we propose Cross Deep Q Network (Cross DQN) to extract the arrangement signal by crossing the embeddings of different items and processing the crossed sequence in the feed. Our model results in higher revenue and better user experience than state-of-the-art baselines in offline experiments. Moreover, our model demonstrates a significant improvement in the online A/B test and has been fully deployed on Meituan feed to serve more than 300 millions of customers.

show abstract

“…And until present, most methods use only vanilla ER, which uniformly samples experiences from the replay buffer. Among them, Zhao et al [39] apply DQN to online recommendation and RNN to generate state embeddings; Chen et al [4] point out that DQN receives unstable rewards in dynamic environments such as online recommendation and may harm the agent; Chen et al [3] found that traditional methods like DQN become intractable when the state becomes higher-dimensional; DPG addresses the intractability by mapping high-dimensional discrete state into low-dimensional continuous state [5,36].…”

Section: Related Workmentioning

confidence: 99%

Locality-Sensitive Experience Replay for Online Recommendation

Chen¹,

Yao²,

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Online recommendation requires handling rapidly changing user preferences. Deep reinforcement learning (DRL) is an effective means of capturing users' dynamic interest during interactions with recommender systems. Generally, it is challenging to train a DRL agent, due to large state space (e.g., user-item rating matrix and user profiles), action space (e.g., candidate items), and sparse rewards. Existing studies leverage experience replay (ER) to let an agent learn from past experience. However, they adapt poorly to the complex environment of online recommender systems and are inefficient in determining an optimal strategy from past experience. To address these issues, we design a novel state-aware experience replay model, which selectively selects the most relevant, salient experiences, and recommends the agent with the optimal policy for online recommendation. In particular, the model uses locality-sensitive hashing to map high dimensional data into low-dimensional representations and a prioritized reward-driven strategy to replay more valuable experience at a higher chance. Experiments on three online simulation platforms demonstrate our model's feasibility and superiority to several existing experience replay methods.

show abstract

Jointly Learning to Recommend and Advertise

Cited by 65 publications

References 21 publications

A Survey of Deep Reinforcement Learning in Recommender Systems: A Systematic Review and Future Directions

A Survey of Deep Reinforcement Learning in Recommender Systems: A Systematic Review and Future Directions

Cross DQN: Cross Deep Q Network for Ads Allocation in Feed

Locality-Sensitive Experience Replay for Online Recommendation

Contact Info

Product

Resources

About