A Universal Approximate Cross-Validation Criterion for Regular Risk Functions

Abstract

Selection of estimators is an essential task in modeling. A general framework is that the estimators of a distribution are obtained by minimizing a function (the estimating function) and assessed using another function (the assessment function). A classical case is that both functions estimate an information risk (specifically cross-entropy); this corresponds to using maximum likelihood estimators and assessing them by Akaike information criterion (AIC). In more general cases, the assessment risk can be estimated by leave-one-out cross-validation. Since leave-one-out cross-validation is computationally very demanding, we propose in this paper a universal approximate cross-validation criterion under regularity conditions (UACVR). This criterion can be adapted to different types of estimators, including penalized likelihood and maximum a posteriori estimators, and also to different assessment risk functions, including information risk functions and continuous rank probability score (CRPS). UACVR reduces to Takeuchi information criterion (TIC) when cross-entropy is the risk for both estimation and assessment. We provide the asymptotic distributions of UACVR and of a difference of UACVR values for two estimators. We validate UACVR using simulations and provide an illustration on real data both in the psychometric context where estimators of the distributions of ordered categorical data derived from threshold models and models based on continuous approximations are compared.

A joint model for multiple dynamic processes and clinical endpoints: Application to Alzheimer's disease

“…Reported are the Akaike and Bayesian information criteria (AIC, BIC) and the universal approximate cross validation criterion (UACV) for dementia and death data conditional on longitudinal information. UACV differences of order 10 −1 , 10 −2 , and 10 −3 qualified as “large,” “moderate,” and “small,” respectively …”

Individualized dynamic prediction of prostate cancer recurrence with and without the initiation of a second treatment: Development and validation

“…This motivated the development of a new estimator of the Brier Score by approximated leave-one-out crossvalidation 16 which is valid and easy to compute on the estimation data.…”

“…Let m be the number of subjects on which the predictive accuracy is computed. It is shown 16 that the difference between 𝒟( A ( θ̂ A ) , B ( θ̂ B )) and Δ( A ( θ̂ A ) , B ( θ̂ B )) is asymptotically normal: $$m^{1/2}\left[\mathrm{scriptD}\left(Afalse({\widehat{\theta }}_{A}false),Bfalse({\widehat{\theta }}_{B}false)\right)-\mathrm{normal\Delta }\left(Afalse({\widehat{\theta }}_{A}false),Bfalse({\widehat{\theta }}_{B}false)\right)\right]\to \mathrm{scriptN}false(0,w_{\ast }^{2}false)$$ where $w_{\ast }^2$ can be estimated by the empirical variance ŵ 2 of the difference of the simple estimators. With z u the u th quantile of a standard normal variable, the confidence interval is then [𝒟 ( A ( θ̂ A ) , B ( θ̂ B )) − z α /2 m −1/2 ŵ ; 𝒟 ( A ( θ̂ A ) , B ( θ̂ B )) + z α /2 m −1/2 ŵ ].…”

“…On the model estimation data, a cross-validation technique is required to correct for overoptimism. With complex models, cross-validation is numerically too expensive to be used, so we applied the approximate leave-one-out cross-validation formula for regular problems proposed by Commenges et al 16 . It is defined as: $$\text{CV}{ℳ}_{\mathrm{a}}false(\hat{\theta },s,\mathrm{scriptT}false)=ℳfalse(\hat{\theta },s,\mathrm{scriptT}false)+N\text{Trace}false(H^{-1}{K}_{s,\mathrm{scriptT}}false)$$ where ℳ( θ̂, s, 𝒯) is the simple estimator (POL, BS or IBS) and the second term is the correction term added by the approximated leave-one-out cross-validation.…”

“…The usual overoptimism due to the use of the same data for the estimation and the prediction was corrected by applying a general formula of approximated cross-validation recently developed 16 . In addition to the Brier Score, we also considered an information criterion for assessing the prognostic value of joint models (EPOCE) 10,17 .…”