Key Concepts in Model Selection: Performance and Generalizability

Abstract

What is model selection? What are the goals of model selection? What are the methods of model selection and how do they work? Which methods perform better than others and in what circumstances? These questions rest on a number of key concepts in a relatively underdeveloped field. The aim of this paper is to explain some background concepts, to highlight some of the results in this special issue, and to add my own. The standard methods of model selection include classical hypothesis testing, maximum likelihood, Bayes method, minimum description length, cross-validation, and Akaike's information criterion. They all provide an implementation of Occam's razor, in which parsimony or simplicity is balanced against goodness-of-fit. These methods primarily take account of the sampling errors in parameter estimation, although their relative success at this task depends on the circumstances. However, the aim of model selection should also include the ability of a model to generalize to predictions in a different domain. Errors of extrapolation, or generalization, are different from errors of parameter estimation. So, it seems that simplicity and parsimony may be an additional factor in managing these errors, in which case the standard methods of model selection are incomplete implementations of Occam's razor. Copyright 2000 Academic Press.

Prognosis of hepatocellular carcinoma

“…In addition, the results of the Cox regression were expressed using the Akaike information criterion, which shows how the explanatory variable (staging systems) affect the dependent variable (survival of HCC)-the lower the Akaike information criterion, the more explanatory it is and the more informative the model is. 22 Lastly, the independent contribution of each staging system to overall prediction of survival in the Cox model was evaluated by comparing the LR test in the full model (all systems included) and in a reduced model when one staging system was removed. 23 All statistical analyses were performed using SAS version 8.1 (Cary, NC), and all graphs were created using MedCalc 7.4 (Mariakerke, Belgium).…”

Selecting an optimal staging system for hepatocellular carcinoma

“…26,27 The lower the AIC, the more explanatory and informative the model is. 28 All statistical analyses were conducted with the SPSS for Windows version 14 release (SPSS, Inc., Chicago, Ill), SAS version 9.1 (SAS, Cary, NJ), and MedCalc for Windows version 4.2 (MedCalc Software, Mariakerke, Belgium). A p value less than 0.05 was considered statistically significant.…”

Analysis of the individual and aggregate genetic contributions of previously identified serine peptidase inhibitor Kazal type 5 (SPINK5), kallikrein-related peptidase 7 (KLK7), and filaggrin (FLG) polymorphisms to eczema risk

“…In any of these approaches we adjusted the models for the common covariates age and gender. 46 Secondly, we defined an interaction score (IC) which counts the number of copies of the potentially disease-associated alleles 47 , which we used as a covariate in the logistic model. Thirdly, we modelled maternal effects observed for SPINK5 using affected offspring from families only and estimated ORs for two different affection status.. compared with controls in a multinomial regression model (MRM), which was carried out with BayesX 1.50 48 .…”