Learning Adaptive Display Exposure for Real-Time Advertising

Abstract

In E-commerce advertising, where product recommendations and product ads are presented to users simultaneously, the traditional setting is to display ads at fixed positions. However, under such a setting, the advertising system loses the flexibility to control the number and positions of ads, resulting in sub-optimal platform revenue and user experience. Consequently, major e-commerce platforms (e.g., Taobao.com) have begun to consider more flexible ways to display ads. In this paper, we investigate the problem of advertising with adaptive exposure: can we dynamically determine the number and positions of ads for each user visit under certain business constraints so that the platform revenue can be increased? More specifically, we consider two types of constraints: requestlevel constraint ensures user experience for each user visit, and platform-level constraint controls the overall platform monetization rate. We model this problem as a Constrained Markov Decision Process with per-state constraint (psCMDP) and propose a constrained two-level reinforcement learning approach to decompose the original problem into two relatively independent sub-problems. To accelerate policy learning, we also devise a constrained hindsight experience replay mechanism. Experimental evaluations on industry-scale real-world datasets demonstrate the merits of our approach in both obtaining higher revenue under the constraints and the effectiveness of the constrained hindsight experience replay mechanism.

Hierarchically Constrained Adaptive Ad Exposure in Feeds

“…Second, constraints for adaptive ad exposure have a hierarchical structure due to the business nature, consisting of request-level constraints (for a single user request) and applicationlevel constraints (for all requests over a period). At the application level, the monetization ratio, indicating the average proportion of ad exposures [23], should be constrained within a specific range. At the fine-grained request level, ad positions are also constrained (e.g., not too high or too dense) for a good user experience [25].…”

Cross DQN: Cross Deep Q Network for Ads Allocation in Feed

“…Such a solution may result in poor differentiation since the PAE of each request is constrained to the same target δ regardless of the context. To allow for differentiation, Wang et al (2019) propose to use an hourlevel constraint that allows for using different PAE targets in different hours. However, the level of differentiation is still limited within an hour.…”

“…• CTLRL (Wang et al 2019). Constrained Two-Level Reinforcement Learning (CTLRL) uses a two-level RL structure to allocate ads.…”

“…PAE is the most important constraint in ads allocation, which balances the user experience and platform revenue. Previous methods constrain all the requests or the requests within the same hour (Wang et al 2019) with the same target PAE, resulting in a lack of personalization and differentiation in the allocation of ads between different requests.…”

Multiplex Behavioral Relation Learning for Recommendation via Memory Augmented Transformer Network

“…Recommender system, which facilitates the information-seeking process of users and meet their personalized interests, have played a critical role in various online services, such as e-commerce systems [13,38], online review platforms [1,44] and advertising [39]. At its core is to learn low-dimensional representations of user-item interaction while capturing the user preference and the underlying intrinsic characteristics [19].…”