Maximum regularized likelihood estimators: A general prediction theory and applications

Abstract

Maximum regularized likelihood estimators (MRLEs) are arguably the most established class of estimators in highdimensional statistics. In this paper, we derive guarantees for MRLEs in the Kullback-Leibler divergence, a general measure of prediction accuracy. We assume only that the densities have a convex parametrization and that the regularization is definite and positive homogenous. The results thus apply to a very large variety of models and estimators, such as tensor regression and graphical models with convex and non-convex regularized methods. A main conclusion is that MRLEs are broadly consistent in prediction-regardless of whether restricted eigenvalues or similar conditions hold.definiteness and positive homogeneity of the regularizers. This makes the results applicable to a variety of parametric and non-parametric models and allows for a broad class of convex and non-convex regularizers.The remainder of this paper is organized as follows. We introduce the framework and the general result in Section 2. We then provide some examples in Section 3. We finally conclude with a brief discussion in Section 4. All proofs are deferred to the Appendices in the supporting information: proofs for the main result to Appendix A, proofs for the examples to Appendix B, and proofs for the bounds of the empirical process terms in Appendix C. In addition, Appendix D contains notation and properties of tensors. Our contributionThe contribution of this work is twofold. First, Theorem 2.1 provides a general prediction guarantee for MRLEs in terms of the Kullback-Leibler loss. Besides being the first assumptionless bound in such a broad setting, the result specializes correctly to known results, such as for lasso, where the corresponding rates have been shown to be optimal up to log-factors. Second, we show that applications of the general theorem to specific examples lead to new guarantees in tensor response regression, generalized linear tensor regression and graphical modelling. The theory thus also establishes new insights into individual cases of MRLEs.

Growing-dimensional partially functional linear models: non-asymptotic optimal prediction error

“…The prediction consistency theory has been well-established for non-parametric and high-dimensional statistics; see Zhuang and Lederer (2018) for the recent development of general regularized maximum likelihood estimators. However, their works mainly aim for non-parametric or high-dimensional models and do not cover the semi-parametric case as studied in our paper.…”

Non-asymptotic Optimal Prediction Error for RKHS-based Partially Functional Linear Models

“…The prediction consistency theory has been well-established for the estimators in non-parametric statistics and high-dimensional statistics, see Zhuang and Lederer (2018) for the recent development for general regularized maximum likelihood estimators. However their works specially aim for nonparametric or high-dimensional models, they do not cover the semi-parametric case as studied in our paper.…”

Thresholded Adaptive Validation: Tuning the Graphical Lasso for Graph Recovery

“…on its off-diagonal. And while there is much theory on the properties of the graphical lasso (Rothman et al, 2008;Ravikumar et al, 2008;Jankova and van de Geer, 2018), as well as general theory for graphical models (Zhuang and Lederer, 2018), there is no theory that includes the practical choice of r.…”