Optimal exponential bounds for aggregation of estimators for the Kullback-Leibler loss

Abstract

We study the problem of model selection type aggregation with respect to the Kullback-Leibler divergence for various probabilistic models. Rather than considering a convex combination of the initial estimators f1, . . . , fN , our aggregation procedures rely on the convex combination of the logarithms of these functions. The first method is designed for probability density estimation as it gives an aggregate estimator that is also a proper density function, whereas the second method concerns spectral density estimation and has no such mass-conserving feature. We select the aggregation weights based on a penalized maximum likelihood criterion. We give sharp oracle inequalities that hold with high probability, with a remainder term that is decomposed into a bias and a variance part. We also show the optimality of the remainder terms by providing the corresponding lower bound results.

Rates of convergence for density estimation with generative adversarial networks

“…The requirement that all functions from D are bounded away from 0 and 1 is needed for the log D and log(1−D) to be well defined. Similar conditions appear in the literature for aggregation with Kullback-Leibler loss (for instance, in Polzehl and Spokoiny [2006], Belomestny and Spokoiny [2007], Rigollet [2012], Butucea et al [2017]).…”

Optimal Kullback–Leibler aggregation in mixture density estimation by maximum likelihood

“…(Catoni, 1997;Yang, 2000;Juditsky et al, 2008) investigated different variants of the progressive mixture rules, also known as mirror averaging (Yuditskiȋ et al, 2005;Dalalyan and Tsybakov, 2012), with respect to the Kullback-Leibler loss and established model selection type oracle inequalities 2 in expectation. Same type of guarantees, but holding with high probability, were recently obtained in (Bellec, 2014;Butucea et al, 2016) for the procedure termed Q-aggregation, introduced in other contexts by (Dai et al, 2012;Rigollet, 2012).…”

Fast adaptive estimation of log-additive exponential models in Kullback-Leibler divergence

“…We take the estimator f θm,n for different values of m ∈ M n , where M n is a sequence of sets of parameter configurations with increasing cardinality. These estimators are not uniformly bounded as required in [11], but we show that they are uniformly bounded in probability and that it does not change the general result. The different values of m correspond to different values of the regularity parameters.…”

“…Notice that the reference probability measure in this paper corresponds to d!1 (x)dx. This implies that ψ λ here differs from the ψ λ of [11] by the constant log(d! ), but this does not affect the calculations.…”

“…Here we extend the convex aggregation scheme for the estimation of the logarithm of the density proposed in [11] to achieve adaptability. We take the estimator f θm,n for different values of m ∈ M n , where M n is a sequence of sets of parameter configurations with increasing cardinality.…”