Analysis and minimization of overtraining effect in rule‐based classifiers for computer‐aided diagnosis

Abstract

Computer-aided diagnostic (CAD) schemes have been developed to assist radiologists detect various lesions in medical images. In CAD schemes, classifiers play a key role in achieving a high lesion detection rate and a low false-positive rate. Although many popular classifiers such as linear discriminant analysis and artificial neural networks have been employed in CAD schemes for reduction of false positives, a rule-based classifier has probably been the simplest and most frequently used one since the early days of development of various CAD schemes. However, with existing rule-based classifiers, there are major disadvantages that significantly reduce their practicality and credibility. The disadvantages include manual design, poor reproducibility, poor evaluation methods such as resubstitution, and a large overtraining effect. An automated rule-based classifier with a minimized overtraining effect can overcome or significantly reduce the extent of the above-mentioned disadvantages. In this study, we developed an "optimal" method for the selection of cutoff thresholds and a fully automated rule-based classifier. Experimental results performed with Monte Carlo simulation and a real lung nodule CT data set demonstrated that the automated threshold selection method can completely eliminate overtraining effect in the procedure of cutoff threshold selection, and thus can minimize overall overtraining effect in the constructed rule-based classifier. We believe that this threshold selection method is very useful in the construction of automated rule-based classifiers with minimized overtraining effect.

Computerized Detection of Lung Nodules in Thin-Section CT Images by Use of Selective Enhancement Filters and an Automated Rule-Based Classifier

“…Please note that the FROC curve for training is located above that for testing, which indicates that overtraining bias occurred in our CAD scheme for nodule detection. However, the extent of overtraining bias in the automated rule-based classifier appears limited because the overtraining bias in the step of cutoff threshold selection was completely eliminated for the automated rule-based classifier [24]. Figure 7 shows the mean FROC curves obtained by testing our CAD scheme for the nodules in the American dataset and the Japanese dataset, and for all nodules.…”

“…The existing rule-based classifiers are generally designed manually, and therefore, they often lead to a large overtraining effect, i.e., a large difference between the estimated performance levels of training and testing [24].…”

“…As most existing rule-based classifiers do, our previous prototypical rulebased classifier employs for a rule only a simple feature, which is either a clinical parameter or a feature directly determined from various medical images [24]. Composite features, which are generally a linear combination of many simple features, may provide a higher performance than does the sequential utilization of individual simple features.…”

“…For this optimal composite feature x, it has been proven that the cutoff threshold must pass through one of the nodules if one wishes to eliminate overtraining in the process of threshold selection [24]. By use of the gain defined below, we developed a technique for automatically determining which nodule the threshold should pass through.…”

“…To achieve a high performance level for a computerized detection scheme for lung nodule, it is important to employ new effective techniques in major steps of the detection scheme, including initial nodule identification and false positive reduction; it is also important to use a relatively large database including both solid nodules and nodules with GGO. In this study, we utilized a selective nodule enhancement filter [23] for improving the detection sensitivity in the initial nodule identification step; we used an automated rule-based classifier [24] for substantial reduction of false positives in the false positive reduction step; and we employed a relatively large database for reliable estimation of the performance level of our CAD scheme. We believe that the two new techniques considerably improved the performance level of our CAD scheme, and that the use of a relatively large database enabled us to estimate the performance level of our CAD scheme reliably.…”