Estimating a difference of Kullback–Leibler risks using a normalized difference of AIC

Abstract

Estimating a difference between Kullback-Leibler risks by a normalized difference of AIC SUMMARY AIC is commonly used for model selection but the precise value of AIC has no direct interpretation. We are interested in quantifying a difference of risks between two models. This may be useful for both an explanatory point of view or for prediction, where a simpler model may be preferred if it does nearly as well as a more complex model. The difference of risks can be interpreted by linking the risks with relative errors in the computation of probabilities and looking at the values obtained for simple models. A scale of values going from negligible to large is proposed. We propose a normalization of a difference of Akaike criteria for estimating the difference of expected Kullback-Leibler risks between maximum likelihood estimators of the distribution in two different models. The variability of this statistic can be estimated. Thus, an interval can be constructed which contains the true difference of expected Kullback-Leibler risks with a pre-specified probability.A simulation study shows that the method works and it is illustrated on two examples. The first is a study of the relationship between body-mass index and depression in elderly people. The second is the choice between models of HIV dynamics, where one model makes the distinction between activated CD4+ T lymphocytes and the other does not.

A Universal Approximate Cross-Validation Criterion for Regular Risk Functions

“…A simple formula allows to estimate these variances and to construct so-called tracking intervals; our simulation study however shows that the coverage of these tracking intervals is too large, due to an overestimation of the variances. This is an open question to find why this happened here while in other contexts [15,19] the coverage rates were correct, and possibly to find a correction to this overestimation; nevertheless, the estimates get the correct order of magnitude and the tracking intervals may be useful.…”

“…However, in the conventional theory of point and interval estimation, the target parameter is fixed; here, it changes with n. Thus, we have a moving target: hence the name of tracking interval. Some simulations in Commenges et al [19] showed that the variance of the difference of AIC was correctly estimated and the corresponding tracking interval had good coverage properties. The same idea can be applied in the more general case treated here.…”

“…Commenges et al [19] using results of Vuong [20] studied the asymptotic distribution of a difference of normalized AIC's as an estimator of a difference of Kullback-Leibler risks: the normalized AIC is defined as 1 2n AIC. Here similar arguments are applied to study the asymptotic distribution of UACVR and a difference of two UACVR values.…”

“…where ω [19] proposed to construct a "tracking interval" for a difference of normalized AIC values. The tracking interval is a kind of confidence interval for the difference of risks.…”

“…The observed distributions of the MMSE sumscore and of its calculation subscore are displayed in Figure 1. Table 4 gives the assessment criteria for estimators based on the different models, and Table 5 [19] suggested to qualify values of order 10 À1 , 10 À2 and 10 À3 as "large," "moderate" and "small," respectively; moreover for multivariate observations, it was suggested to divide by the total number of observations rather by the number of independent observations. With this correction (which amounts to divide the current values by a factor of 3:7 ¼ 10; 846=2; 914) the differences between the linear model and the other models can be qualified as "large," and the differences between the threshold model and both beta c.d.f.…”