Conditional confidence intervals for classification error rate

Adaptive Confidence Intervals for the Test Error in Classification

“…Then the 1 − γ CPB interval is given by $[\widehat{\tau }\left(\hat{c}\right)-\widehat{u}∕\sqrt{n},\widehat{\tau }\left(\hat{c}\right)-\widehat{l}∕\sqrt{n}]$. The second approach is based on the asymptotic approximation $$\sqrt{n}({\mathbb{P}}_n-P)1_{Y{X}^t{\hat{\beta }}_n<0}\approx N(0,(1-P{1}_{Y{X}^t{\beta }^{\ast }<0})P{1}_{Y{X}^t{\beta }^{\ast }<0}).$$ Thus the normal approximation confidence set is given by $\widehat{\tau }\left(\hat{c}\right)\pm z_{1-\gamma }\sqrt{\frac{\hat{\tau }\left(\widehat{c}\right)(1-\widehat{\tau }\left(\hat{c}\right))}{n}}$ (see the binomial approximation in Chung and Han 2009). If P ( X t β * = 0) = 0 then both methods can be shown to be consistent.…”

“…The test error $\tau \left(\hat{c}\right)$ is a functional of $\hat{c}$ and thus is a random quantity. For this reason $\tau \left(\hat{c}\right)$ is sometimes referred to as the conditional test error (Efron 1997; Hastie et al 2009; Chung and Han 2009). Estimation of the test error typically employs resampling.…”

“…Yang (2006) follows this paradigm as well, using a normal approximation to the repeated split cross-validation estimator. In practice, the point estimation paradigm is often applied by simply bootstrapping the estimator of the test error (see Jiang et al, 2008; Chung and Han 2009). These methods, while intuitive, lack theoretical justification.…”

Structural, Conformational and Interactional Investigation of Proteins with Related Sequences and Multiple Structures

“…The five linear relations p i,j ((4(a)-(e)) ( Figure 3)) are dependent only of primary sequences (by n L , n H and N), and they will be validated by predictions in homologous protein samples similar to cross-validation assays in statistics [34]; though here is focused on a thorough case study by means of a rule-based approach ((3), (4(a)-(e))), so that it does not suffice to identify the occurrence and to determine the quantity of a type of mechanism, the protein names (PDB ID) should be precisely furnished whenever necessary. Furthermore, the four types of molecular mechanisms and their amounts should be confirmed, complemented or denied in the following more precise inspections for another detached sample set, the homologous proteins.…”