Information-theoretic determination of minimax rates of convergence

Yang, Yuhong; Barron, Andrew R.

doi:10.1214/aos/1017939142

Cited by 461 publications

(380 citation statements)

References 53 publications

Supporting

Mentioning

370

Contrasting

Unclassified

Order By: Relevance

“…, β M }, and N 2 ( ) denote the minimal cardinality of an -covering of B q (R q ) in 2 -norm. We follow the standard technique in [30] to transform the estimation on lower bound into a multi-way hypothesis testing problem as follows…”

Section: Resultsmentioning

confidence: 99%

See 1 more Smart Citation

Minimax Rates of ℓp-Losses for High-Dimensional Linear Errors-in-Variables Models over ℓq-Balls

2021

Entropy

View full text Add to dashboard Cite

In this paper, the high-dimensional linear regression model is considered, where the covariates are measured with additive noise. Different from most of the other methods, which are based on the assumption that the true covariates are fully obtained, results in this paper only require that the corrupted covariate matrix is observed. Then, by the application of information theory, the minimax rates of convergence for estimation are investigated in terms of the ℓp(1≤p<∞)-losses under the general sparsity assumption on the underlying regression parameter and some regularity conditions on the observed covariate matrix. The established lower and upper bounds on minimax risks agree up to constant factors when p=2, which together provide the information-theoretic limits of estimating a sparse vector in the high-dimensional linear errors-in-variables model. An estimator for the underlying parameter is also proposed and shown to be minimax optimal in the ℓ2-loss.

show abstract

Section: Resultsmentioning

confidence: 99%

“…andβ is an estimator taking values in the packing set. It then follows from Fano's inequality [30] that…”

Section: Resultsmentioning

confidence: 99%

Minimax Rates of ℓp-Losses for High-Dimensional Linear Errors-in-Variables Models over ℓq-Balls

2021

Entropy

View full text Add to dashboard Cite

show abstract

“…Considering the well developed body of theory for bounding and/or computing the minimax risk for various statistical estimation problems, for example, see Yang and Barron (1999) and references therein, deriving the optimal minimax rate for estimators under the distributed inference setting will be an interesting future research direction.…”

Section: Theoretical Results On the Distributed Sparse Estimatormentioning

confidence: 99%

Aggregated inference

Huo

Cao

2018

WIREs Computational Stats

View full text Add to dashboard Cite

Aggregated inference on distributed data becomes more and more important due to the larger size of data collected in different industries. Modeling and inference are needed in the case where data cannot be obtained at a central location; aggregated statistical inference is a major tool to solve the aforementioned problems. In the literature, problems under the setting of regression model (more generally, M‐estimator) are extensively studied. There are at least two popular techniques for distributed estimation: (a) averaging estimators from local locations and (b) the one‐step approach, which combines the simple averaging estimator with a classical Newton's method (using the local Hessian matrices) to generate a “one‐step” estimator. It is proved that under certain assumptions, the above constructed estimators enjoy the same asymptotic properties as the centralized estimator, which is obtained as if all data were available at a central location. We review the aforementioned two major estimations. It can be seen that, in Big‐Data problems, dividing the data to multiple machines and then using the aggregation technique to solve the estimation problem in parallel can speed up the computation with little compromise of the quality of the estimators. We discuss potential extensions to other models, such as support vector machine, principle component analysis, and so on. Numerical examples are omitted due to the space limitation; they can be easily found in the literature. This article is categorized under: Statistical Learning and Exploratory Methods of the Data Sciences > Knowledge Discovery Statistical Learning and Exploratory Methods of the Data Sciences > Modeling Methods Statistical Models > Fitting Models Statistical and Graphical Methods of Data Analysis > Modeling Methods and Algorithms

show abstract

“…It has been shown that increasing mutual information implies decreasing estimation error for a wide variety of definitions of statistical error and loss criteria (e.g., Guntuboyina, 2011;Palomar & Verdu, 2007;Yang & Barron, 1999). As such, for a very broad set of models and error criteria, choosing measures so as to increase the generalized reliability (Equation 1) will increase measurement precision and decrease estimation error.…”

Section: Relationships Between Lindley Information and Estimation Errormentioning

confidence: 99%

Reliability as Lindley Information

Ke¹

2017

Preprint

View full text Add to dashboard Cite

This paper introduces a generalized definition of reliability based on Lindley information, which is the mutual information between an observed measure and latent attribute. This definition reduces to the traditional definition of reliability in the case of normal variables, but can be applied to any joint distribution of observed and latent variables. Importantly, unlike traditional definitions of reliability, this formulation of reliability applies to vector-or matrix-valued estimates and summaries of responses, and therefore generalizes reliability to sets of scores and estimates in addition to individual scores and estimates. This formulation also leads to new bounds for reliability, as well as newly reported relationships between reliability and the traditional Fisher information function familiar in item response theory literature. The generalized reliability can be estimated using formulae, or methods used in Bayesian inference such as Markov Chain Monte Carlo (MCMC) depending on the case. Examples based on well-studied datasets are provided, as well as an application to randomly-varying intraindividual covariance structures.Running head: GENERALIZED RELIABILITY 3 The aim of this paper is to introduce a generalized definition of reliability given aswhere ι is the Lindley information (Lindley, 1956), which in this context is the mutual information between an observed variable X and a latent attribute θ:Either X or θ could be multidimensional.Lindley information can be interpreted various ways depending on inferential paradigm and perspective: as an index of the expected change in probability of X and θ under knowledge of their joint distribution, relative to knowledge only of their independent marginal distributions; as the expected change in probability of θ given the data, relative to the prior distribution of θ; as the expected per-observation log-likelihood ratio comparing a model of measure informativeness to a model of measure uninformativeness (Kinney & Atwal, 2014); or as the difference in codelength required to describe the data and θ under a model of informativeness compared to a model of uninformativeness.More generally, Lindley information can be interpreted as the divergence of the joint observed-latent distribution p(X, θ) from the independence distribution p(X)p(θ). As such, it is a special case of f-divergence (Amari, 2009;Gilardoni, 2010;Liese & Vajda, 2006;Sason & Verdu, 2016), which can be defined aswhere f(u) is a convex function f:(0, ∞) → , and p and q are probability densities absolutely continuous with ℝ regard to a reference measure ν. Lindley information is obtained by setting f(u) = u ln u, with p = p(X, θ) and q = p(X)p(θ), and is equivalent to the relative entropy or Kullback-Liebler divergence between p(X, θ) and p(X)p(θ).Like traditional reliability, the generalized definition of reliability implied by Equations 1 and 2 produces a statistic that generally ranges between 0 and 1, and reflects the measurement precision of an indicator, such that values closer to 1 indic...

show abstract

Information-theoretic determination of minimax rates of convergence

Cited by 461 publications

References 53 publications

Minimax Rates of ℓp-Losses for High-Dimensional Linear Errors-in-Variables Models over ℓq-Balls

Minimax Rates of ℓp-Losses for High-Dimensional Linear Errors-in-Variables Models over ℓq-Balls

Aggregated inference

Reliability as Lindley Information

Contact Info

Product

Resources

About