Distributional Random Forests: Heterogeneity Adjustment and Multivariate Distributional Regression

Ćevid, Domagoj; Michel, Loris; Näf, Jeffrey; Meinshausen, Nicolai; Bühlmann, Peter

doi:10.48550/arxiv.2005.14458

“…This implies that the continuum estimator proposed in Section 4.1, as a convex combination of Lipschitz functions, is also Lipschitz. Point-wise rates provided in [ Ćevid et al, 2020, Oprescu et al, 2019, Athey et al, 2019, Györfi et al, 2002 can then be translated into uniform rates, thanks to this Lipschitz property (see Lemma D.10).…”

Section: Regret Guaranteesmentioning

confidence: 99%

“…Here, ω i (s, a) roughly measures the proximity of the i th datapoint to the query point (s, a), so it is typically larger when (s i , a i ) is closer to (s, a). Common weight construction methods include k-nearest neighbors, kernel regressions, decision trees and various tree ensembles [Bertsimas and Kallus, 2020, Ćevid et al, 2020, Khosravi et al, 2019, Oprescu et al, 2019, Meinshausen and Ridgeway, 2006, Athey et al, 2019. With these weights, we can approximate f 0 (s, a; α) for any α with the following continuum estimator:…”

Section: Estimating a Continuum Of Nuisancesmentioning

confidence: 99%

Doubly Robust Distributionally Robust Off-Policy Evaluation and Learning

Kallus¹,

Mao²,

Wang³

et al. 2022

Preprint

0

View full text Add to dashboard Cite

Off-policy evaluation and learning (OPE/L) use offline observational data to make better decisions, which is crucial in applications where experimentation is necessarily limited. OPE/L is nonetheless sensitive to discrepancies between the data-generating environment and that where policies are deployed. Recent work proposed distributionally robust OPE/L (DROPE/L) to remedy this, but the proposal relies on inverse-propensity weighting, whose regret rates may deteriorate if propensities are estimated and whose variance is suboptimal even if not. For vanilla OPE/L, this is solved by doubly robust (DR) methods, but they do not naturally extend to the more complex DROPE/L, which involves a worst-case expectation. In this paper, we propose the first DR algorithms for DROPE/L with KL-divergence uncertainty sets. For evaluation, we propose Localized Doubly Robust DROPE (LDR 2 OPE) and prove its semiparametric efficiency under weak product rates conditions. Notably, thanks to a localization technique, LDR 2 OPE only requires fitting a small number of regressions, just like DR methods for vanilla OPE. For learning, we propose Continuum Doubly Robust DROPL (CDR 2 OPL) and show that, under a product rate condition involving a continuum of regressions, it enjoys a fast regret rate of O(N −1/2 ) even when unknown propensities are nonparametrically estimated. We further extend our results to general f -divergence uncertainty sets. We illustrate the advantage of our algorithms in simulations.

show abstract

“…Chagny [17] used an expansion of a "warped" conditional density onto a space spanned by orthonormal bases. Recently, Ćevid et al [18] studied conditional density estimation using an adapted Random Forest algorithm. In conclusion, although there has been some work on conditional density estimation, the minimax optimal rate is an open question.…”

Section: Relevant Literaturementioning

confidence: 99%

Minimax optimal conditional density estimation under total variation smoothness

Neykov

¹

,

Balakrishnan

²

2022

Electron. J. Statist.

3

0

View full text Add to dashboard Cite

This paper studies the minimax rate of nonparametric conditional density estimation under a weighted absolute value loss function in a multivariate setting. We first demonstrate that conditional density estimation is impossible if one only requires that p X|Z is smooth in x for all values of z. This motivates us to consider a sub-class of absolutely continuous distributions, restricting the conditional density p X|Z (x|z) to not only be Hölder smooth in x, but also be total variation smooth in z. We propose a corresponding kernel-based estimator and prove that it achieves the minimax rate. We give some simple examples of densities satisfying our assumptions which imply that our results are not vacuous. Finally, we propose an estimator which achieves the minimax optimal rate adaptively, i.e., without the need to know the smoothness parameter values in advance. Crucially, both of our estimators (the adaptive and non-adaptive ones) impose no assumptions on the marginal density p Z , and are not obtained as a ratio between two kernel smoothing estimators which may sound like a go to approach in this problem.

show abstract

“…Chagny [18] used an expansion of a "warped" conditional density onto a space spanned by orthonormal bases. Recently, Ćevid et al [19] studied conditional density estimation using an adapted Random Forest algorithm. In conclusion, although there has been some work on conditional density estimation, the minimax optimal rate is an open question.…”

Section: Relevant Literaturementioning

confidence: 99%

Minimax Optimal Conditional Density Estimation under Total Variation Smoothness

Neykov¹,

Balakrishnan²

2021

Preprint

View full text Add to dashboard Cite

This paper studies the minimax rate of nonparametric conditional density estimation under a weighted absolute value loss function in a multivariate setting. We first demonstrate that conditional density estimation is impossible if one only requires that p X|Z is smooth in x for all values of z. This motivates us to consider a sub-class of absolutely continuous distributions, restricting the conditional density p X|Z (x|z) to not only be Hölder smooth in x, but also be total variation smooth in z. We propose a corresponding kernel-based estimator and prove that it achieves the minimax rate. We give some simple examples of densities satisfying our assumptions which imply that our results are not vacuous. Finally, we propose an estimator which achieves the minimax optimal rate adaptively, i.e., without the need to know the smoothness parameter values in advance. Crucially, both of our estimators (the adaptive and non-adaptive ones) impose no assumptions on the marginal density p Z , and are not obtained as a ratio between two kernel smoothing estimators which may sound like a go to approach in this problem.

show abstract

Distributional Random Forests: Heterogeneity Adjustment and Multivariate Distributional Regression

Cited by 4 publications

References 12 publications

Doubly Robust Distributionally Robust Off-Policy Evaluation and Learning

Doubly Robust Distributionally Robust Off-Policy Evaluation and Learning

Minimax optimal conditional density estimation under total variation smoothness

Minimax Optimal Conditional Density Estimation under Total Variation Smoothness

Contact Info

Product

Resources

About