Estimation in Gaussian Noise: Properties of the Minimum Mean-Square Error

Guo, Dongning; Wu, Yihong; Shamai, Shlomo; Verdú, S.

doi:10.1109/tit.2011.2111010

Cited by 211 publications

(12 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Each inequality in Lemma 2 is satisfied with equality if and only if the input X is Gaussian. That is, for Gaussian inputs, Var(X|Y ) is constant almost surely, and mmse and differential entropy are maximized [21]. From (29), it is evident that the maximum differential entropy of the conditional mean is achieved when the input X is Gaussian even though it minimizes the variance of E[X|Y ], which we record in the following corollary:…”

Section: A Upper Boundsmentioning

confidence: 75%

“…1. Properties of MMSE such as monotonicity, convexity, and infinite differentiability as a function of snr have been shown in [21], while its functional properties as a function of inputoutput distribution have been analyzed in [22], [23]. Recently, in [24], [25], the authors have focused on the derivatives of the conditional mean with respect to the observation, and many previously known identities in the literature have been recovered.…”

Section: B Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Differential Entropy of the Conditional Expectation Under Additive Gaussian Noise

Atalik

Köse

Gastpar

2022

IEEE Trans. Signal Process.

View full text Add to dashboard Cite

The conditional mean is a fundamental and important quantity whose applications include the theories of estimation and rate-distortion. It is also notoriously difficult to work with. This paper establishes novel bounds on the differential entropy of the conditional mean in the case of finite-variance input signals and additive Gaussian noise. The main result is a new lower bound in terms of the differential entropies of the input signal and the noisy observation. The main results are also extended to the vector Gaussian channel and to the natural exponential family. Various other properties such as upper bounds, asymptotics, Taylor series expansion, and connection to Fisher Information are obtained. Two applications of the lower bound in the remote-source coding and CEO problem are discussed.

show abstract

Section: A Upper Boundsmentioning

confidence: 75%

Section: B Related Workmentioning

confidence: 99%

Differential Entropy of the Conditional Expectation Under Additive Gaussian Noise

Atalik

Köse

Gastpar

2022

IEEE Trans. Signal Process.

View full text Add to dashboard Cite

show abstract

“…Existing approaches for noise variance estimation equate the mean value in (60) with the available sample r MS m,n and provide the following noise variance estimate [51,52]:…”

Section: Noise Variance Estimation When θ M * ∈ H Mmentioning

confidence: 99%

Learnability, Sample Complexity, and Hypothesis Class Complexity for Regression Models

Beheshti¹,

Shamsi²

2023

Preprint

View full text Add to dashboard Cite

The goal of a learning algorithm is to receive a training data set as input and provide a hypothesis that can generalize to all possible data points from a domain set. The hypothesis is chosen from hypothesis classes with potentially different complexities. Linear regression modeling is an important category of learning algorithms. The practical uncertainty of the target samples affects the generalization performance of the learned model. Failing to choose a proper model or hypothesis class can lead to serious issues such as underfitting or overfitting. These issues have been addressed by alternating cost functions or by utilizing cross-validation methods. These approaches can introduce new hyperparameters with their own new challenges and uncertainties or increase the computational complexity of the learning algorithm. On the other hand, the theory of probably approximately correct (PAC) aims at defining learnability based on probabilistic settings. Despite its theoretical value, PAC does not address practical learning issues on many occasions. This work is inspired by the foundation of PAC and is motivated by the existing regression learning issues. The proposed approach, denoted by ǫ-Confidence Approximately Correct (ǫ-CoAC), utilizes Kullback-Leibler divergence (relative entropy) and proposes a new related typical set in the set of hyperparameters to tackle the learnability issue. ǫ-CoAC learnability is able to validate the learning process as a function of sample size as well as a function of the hypothesis class complexity order. Moreover, it enables the learner to compare hypothesis classes of different complexity orders and choose among them the optimum with the minimum ǫ in the ǫ-CoAC framework. Not only the ǫ-CoAC learnability overcomes the issues of overfitting and underfitting, but it also shows advantages and superiority over the well-known cross-validation method in the sense of time consumption as well as in the sense of accuracy.

show abstract

“…where Equation (25a) holds by substituting t i witht i in Equation 20, Equation (25b) is due to the optimality of t i at the next iteration, and Equation (25c) is due to the concavity of T k,i w.r.t. t i [46]. Hence, according to [47], the proposed SCA algorithm will converge to a point leading to a locally optimal solution of Equation (17), which satisfies KKT conditions of the original Equation (17).…”

Section: Update Of P Ki With Given σ 2 Aimentioning

confidence: 99%

Joint Resource Allocation for Frequency-Domain Artificial Noise Assisted Multiuser Wiretap OFDM Channels with Finite-Alphabet Inputs

et al. 2019

View full text Add to dashboard Cite

The security issue on the physical layer is of significant challenge yet of paramount importance for 5G communications. In some previous works, transmit power allocation has already been studied for orthogonal frequency division multiplexing (OFDM) secure communication with Gaussian channel inputs for both a single user and multiple users. Faced with peak transmission power constraints, we adopt discrete channel inputs (e.g., equiprobable Quadrature Phase Shift Keying (QPSK) with symmetry) in a practical communication system, instead of Gaussian channel inputs. Finite-alphabet inputs impose a more significant challenge as compared with conventional Gaussian random inputs for the multiuser wiretap OFDM systems. This paper considers the joint resource allocation in frequency-domain artificial noise (AN) assisted multiuser wiretap OFDM channels with discrete channel inputs. This security problem is formulated as nonconvex sum secrecy rate optimization by jointly optimizing the subcarrier allocation, information-bearing power, and AN-bearing power. To this end, with a suboptimal subcarrier allocation scheme, we propose an efficient iterative algorithm to allocate the power between the information and the AN via the Lagrange duality method. Finally, we carry out some numerical simulations to demonstrate the performance of the proposed algorithm.

show abstract

Estimation in Gaussian Noise: Properties of the Minimum Mean-Square Error

Cited by 211 publications

References 37 publications

Differential Entropy of the Conditional Expectation Under Additive Gaussian Noise

Differential Entropy of the Conditional Expectation Under Additive Gaussian Noise

Learnability, Sample Complexity, and Hypothesis Class Complexity for Regression Models

Joint Resource Allocation for Frequency-Domain Artificial Noise Assisted Multiuser Wiretap OFDM Channels with Finite-Alphabet Inputs

Contact Info

Product

Resources

About