A Markov Decision Model for Managing Display-Advertising Campaigns

Abstract

Problem definition: Managers in ad agencies are responsible for delivering digital ads to viewers on behalf of advertisers, subject to the terms specified in the ad campaigns. They need to develop bidding policies to obtain viewers on an ad exchange and allocate them to the campaigns to maximize the agency’s profits, subject to the goals of the ad campaigns. Academic/practical relevance: Determining a rigorous solution methodology is complicated by uncertainties in the arrival rates of viewers and campaigns, as well as uncertainty in the outcomes of bids on the ad exchange. In practice, ad hoc strategies are often deployed. Our methodology jointly determines optimal bidding and viewer-allocation strategies and obtains insights about the characteristics of the optimal policies. Methodology: New ad campaigns and viewers are treated as Poisson arrivals, and the resulting model is a Markov decision process, where the state of the system is the number of undelivered impressions in queue for each campaign type in each period. We develop solution methods for bid optimization and viewer allocation and perform a sensitivity analysis with respect to the key problem parameters. Results: We solve for the optimal dynamic, state-dependent bidding and allocation policies as a function of the number of ad impressions in queue, for both the finite horizon and steady-state cases. We show that the resulting optimization problems are strictly concave in the decision variables and develop and evaluate a heuristic method that can be applied to large problems. Managerial implications: Numerical analysis of our heuristic solution shows that its errors are generally small and that the optimal dynamic, state-dependent bidding policies obtained by our model are significantly better than optimal static policies. Our proposed approach is managerially attractive because it is easy to implement in practice. We identify the capacity of the impression queue as an important managerial control lever and show that it can be more effective than using higher bids to reduce delay penalties. We quantify potential operational benefits from the consolidation of ad campaigns, as well as merging ad exchanges. Supplemental Material: The online appendix is available at https://doi.org/10.1287/msom.2022.1142 .

Improving Real-Time Bidding in Online Advertising using Markov Decision Processes and Machine Learning Techniques

“…Du et al [17] propose a constrained Markov decision process (CMDP) approach for enhancing real-time bidding, considering ad impression quality, budget constraints, and bidding strategy. Agrawal et al [18] present a Markov decision model for administering displayadvertising campaigns that employ a dynamic pricing mechanism to calculate optimal bid amounts. Shanahan and den Poel [19] propose a Markov decision process approach to determine online advertisements' optimal frequency limitation policy to maximise click-through rates.…”