Environment Reconstruction with Hidden Confounders for Reinforcement Learning based Recommendation

Abstract

Reinforcement learning aims at searching the best policy model for decision making, and has been shown powerful for sequential recommendations. The training of the policy by reinforcement learning, however, is placed in an environment. In many real-world applications, however, the policy training in the real environment can cause an unbearable cost, due to the exploration in the environment. Environment reconstruction from the past data is thus an appealing way to release the power of reinforcement learning in these applications. The reconstruction of the environment is, basically, to extract the casual effect model from the data. However, real-world applications are often too complex to offer fully observable environment information. Therefore, quite possibly there are unobserved confounding variables lying behind the data. The hidden confounder can obstruct an effective reconstruction of the environment. In this paper, by treating the hidden confounder as a hidden policy, we propose a deconfounded multi-agent environment reconstruction (DEMER) approach in order to learn the environment together with the hidden confounder. DEMER adopts a multi-agent generative adversarial imitation learning framework. It proposes to introduce the confounder embedded policy, and use the compatible discriminator for training the policies. We then apply DEMER in an application of driver program recommendation. We firstly use an artificial driver program recommendation environment, abstracted from the real application, to verify and analyze the effectiveness of DEMER. We then test DEMER in the real application of Didi Chuxing. Experiment results show that DEMER can effectively reconstruct the hidden confounder, and thus can build the environment better. DEMER also derives a recommendation policy with

A Survey on Deep Reinforcement Learning for Data Processing and Analytics

“…However, the recommendation often fails when drivers make decisions in a complex environment. To address this issue, in [111] a new method is proposed to model hidden causal factors, called confounders, in a complex environment. Specifically, the framework in [50] is extended to include the confounders.…”

“…This presents difficulty in evaluating the methods for different environments. For example, the confounders modeling hidden causal factors in [111] can also contribute to DRL modeling in E-commerce. This is because modeling customers' interests are always subject to changing environments and a new environment may contain hidden causal factors.…”

Choosing the Best of Both Worlds

“…In the off-policy setting, Chen et al [5] and Zhao et al [45] proposed the use of propensity scores to perform off-policy correction, but with training difficulties due to high-variance. Model-based RL approaches [6,34,47] first build a model to simulate the environment, in order to avoid any issues with off-policy training. However, these two-stage approaches depend heavily on the accuracy of the simulator.…”

Supervised Advantage Actor-Critic for Recommender Systems

“…However, the estimation of propensity scores has high variance and there are tricks like smoothing or clipping to train the model (we discuss this in section 4). Model-based RL approaches [2,33,47] attempt to eliminate the off-policy issue by building a model to simulate the environment. The policy can then be trained through interactions with the simulator.…”