Low-Complexity Methods for Estimation After Parameter Selection

Abstract

Statistical inference of multiple parameters often involves a preliminary parameter selection stage. The selection stage has an impact on subsequent estimation, for example by introducing a selection bias. The post-selection maximum likelihood (PSML) estimator is shown to reduce the selection bias and the post-selection mean-squared-error (PSMSE) compared with conventional estimators, such as the maximum likelihood (ML) estimator. However, the computational complexity of the PSML is usually high due to the multi-dimensional exhaustive search for a global maximum of the post-selection log-likelihood (PSLL) function. Moreover, the PSLL involves the probability of selection that, in general, does not have an analytical form. In this paper, we develop new low-complexity post-selection estimation methods for a two-stage estimation after parameter selection architecture. The methods are based on implementing the iterative maximization by parts (MBP) approach, which is based on the decomposition of the PSLL function into "easilyoptimized" and complicated parts.The proposed second-best PSML method applies the MBP-PSML algorithm with a pairwise probability of selection between the two highest-ranked parameters w.r.t. the selection rule. The proposed SA-PSML method is based on using stochastic approximation (SA) and Monte Carlo integrations to obtain a non-parametric estimation of the gradient of the probability of selection and then applying the MBP-PSML algorithm on this approximation. For low-complexity performance analysis, we develop the empirical post-selection Cramér-Rao-type lower bound. Simulations demonstrate that the proposed post-selection estimation methods are tractable and reduce both the bias and the PSMSE, compared with the ML estimator, while only requiring moderate computational complexity.

Non-Bayesian Parametric Missing-Mass Estimation

“…1) mmCCRB on the mmMSE of missing-mass unbiased estimators: For the sake of simplicity of derivation, we assume in this subsection that b N,0 (θ) = 0. According to Proposition 1, this condition is a sufficient condition for the Lehmann unbiasedness in (32) and (16). For this case and the classical model described by (3), it can be verified that S(θ) from ( 38) is a diagonal matrix with the diagonal elements…”

“…In particular, the CCRB [26][27][28][29], which is associated with the CML estimator, is unsuited as a bound on the performance of Good-Turing estimators outside the asymptotic region, while it provides a lower bound on the MSE of any χ-unbiased estimator [28][29][30]. Our recent works on non-Bayesian estimation after selection [31][32][33] suggest that conditional schemes, in which the performance criterion depends on the observed data, require different CRB-type bounds.…”

“…The measure of closeness is determined by the considered cost function, C( θ, θ). Examples for Lehmann unbiasedness with different cost functions and under parametric constraints can be found in [28,29,[31][32][33][34]43].…”

“…It should be noted that the condition in (31) requires a uniform unbiasedness, for any θ ∈ Ω θ , while the conditions in (32) and (33) are local conditions that are required to be satisfied only at the specific θ, denoted here by θ, for which the bound is developed. Both the local and uniform missingmass unbiasedness definitions restrict only the values that belong to the set of unseen symbols, i.e.…”

Cramer-Rao Bound for Estimation After Model Selection and its Application to Sparse Vector Estimation

“…However, in [38], [39], [40], [41], the useful data is selected and not the model. In [42], [43], [44], [45], we developed the CRB and estimation methods for models whose "parameters of interest" are selected based on the data, i.e. estimation after parameter selection, in which the model is perfectly known.…”