Policy-Adaptive Estimator Selection for Off-Policy Evaluation

Doubly Robust Off-Policy Evaluation for Ranking Policies under the Cascade Behavior Model

“…We thus consider the MSE estimation task as an independent research topic and do not propose specific approaches to estimate the MSE from the logged data. Instead, our experiments will demonstrate that AIPS with our data-driven procedure for behavior model optimization performs reasonably well across a variety of experiment settings, even with a noisy MSE estimate and with an MSE estimated via an existing method from Udagawa et al [37] that uses only the observed logged data.…”

“…Our work also raises several intriguing research questions for future studies. First, it would be valuable to develop an accurate way to estimate the MSE of an OPE estimator beyond existing methods [35,37] to better optimize the user behavior model to further improve AIPS. Second, OPE of ranking policies can still become extremely difficult when the number of unique actions (|A|) is large.…”

“…It is important to note that our algorithm is agnostic to the method used for estimating the MSE. For example, we can estimate the MSE by following existing methods from [35] or [37]. We thus consider the MSE estimation task as an independent research topic and do not propose specific approaches to estimate the MSE from the logged data.…”

“…AIPS uses C = {𝒄 𝑆 , 𝒄 𝐶 , 𝒄 𝐼 , 𝒄 ′ , 𝒄 ′′ } as the candidate set of behavior models where 𝒄 𝑆 , 𝒄 𝐶 , and 𝒄 𝐼 are defined in Section 4, and 𝒄 ′ and 𝒄 ′′ are defined in Appendix A. When performing user behavior optimization of AIPS, its MSE is estimated by PAS-IF [37] using only the observable logged data. Note that we cannot implement AIPS (true) in this section, since we do not know the true user behavior in the real-world dataset.…”