A Review of Caveats in Statistical Nuclear Image Analysis

Abstract

A large body of the published literature in nuclear image analysis do not evaluate their findings on an independent data set. Hence, if several features are evaluated on a limited data set over‐optimistic results are easily achieved. In order to find features that separate different outcome classes of interest, statistical evaluation of the nuclear features must be performed. Furthermore, to classify an unknown sample using image analysis, a classification rule must be designed and evaluated. Unfortunately, statistical evaluation methods used in the literature of nuclear image analysis are often inappropriate. The present article discusses some of the difficulties in statistical evaluation of nuclear image analysis, and a study of cervical cancer is presented in order to illustrate the problems. In conclusion, some of the most severe errors in nuclear image analysis occur in analysis of a large feature set, including few patients, without confirming the results on an independent data set. To select features, Bonferroni correction for multiple test is recommended, together with a standard feature set selection method. Furthermore, we consider that the minimum requirement of performing statistical evaluation in nuclear image analysis is confirmation of the results on an independent data set. We suggest that a consensus of how to perform evaluation of diagnostic and prognostic features is necessary, in order to develop reliable tools for clinical use, based on nuclear image analysis.

Prognostic Value of Graph Theory-Based Tissue Architecture Analysis in Carcinomas of the Tongue

“…For a given set of possible but not established prognostic markers, re-substitution of cases in a data set and cross-validation of classification performance tends to give overly optimistic results (Schulerud et al, 1998). Over-fitting (the structural features give a good characterization of the class in the specific learning set but not in the general population of cases to be investigated) may occur if the number of patients in the smallest outcome group is very different from that in the largest group, or if the number of structural features is large compared with the number of cases investigated.…”

High‐resolution image cytometry on smears of normal oral mucosa: a possible approach for the early detection of laryngopharyngeal cancers

“…If one does not stick to these canons, the danger of overoptimistic results is therefore very high. 26 In the study presented, these rules were respected. Selection of cases was done randomly for both learning and test set.…”

“…26 Whenever using multivariate analysis, the most important consideration is a reasonable ratio of the sample size in relation to the number of features integrated in the classification rule. 31 -33 When many features are used, an independent test set is necessary to evaluate the performance of the classifier.…”

“…To avoid overtraining, a maximum of two features was allowed to build the discriminant function with respect to the number of cases included in the study. 26 To evaluate the performance of the classifier, the same discriminant function was applied to the independent test set.…”

Quantitative nuclear texture features analysis confirms WHO classification 2004 for lung carcinomas

“…To avoid overtraining, a maximum of four features was allowed to create a discriminant function with respect to the number of cases included in the study. 23 The discriminant function derived from the tumor classes described above was then applied to each single tumor entity. The classifying power of this discriminant function was expressed by the percentage of overall correctly classified cases for each tumor class (compare cross tables in the 'Results' section).…”