Comparison of Non-Parametric Methods for Assessing Classifier Performance in Terms of ROC Parameters

Yousef, Waleed A.; Wagner, Robert F.; Loew, Murray H.

doi:10.1109/aipr.2004.18

Cited by 26 publications

(20 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To manually select an ␣ value, we trained ANNs with ␣ values ranging between 0.01 and 4.0, repeated the experiment independently 50 times, and chose the ␣ value that maximized the .632ϩ bootstrapping AUC value ͑calculated on the training dataset only͒ averaged across the replicated experiments. 29,30 This ␣ value was then used as a fixed value in all ANN training experiments. We used the method of .632ϩ bootstrapping because we did not want to bias the results by involving the validation dataset during training.…”

Section: Iie Banns and Weight Decaymentioning

confidence: 99%

“…However, withholding training cases for testing is not an efficient use of the data for small training datasets. We used the method of .632ϩ bootstrapping, 7,29,30 which allows all cases be used for training. Figure 2͑a͒ shows how the AUC values of ANNs varied with training iterations for three testing methods that could be used to determine when to stop training for the XOR experiment.…”

Section: Iif Early Stoppingmentioning

confidence: 99%

See 1 more Smart Citation

Noise injection for training artificial neural networks: A comparison with weight decay and early stopping

et al. 2009

View full text Add to dashboard Cite

The purpose of this study was to investigate the effect of a noise injection method on the "overfitting" problem of artificial neural networks (ANNs) in two-class classification tasks. The authors compared ANNs trained with noise injection to ANNs trained with two other methods for avoiding overfitting: weight decay and early stopping. They also evaluated an automatic algorithm for selecting the magnitude of the noise injection. They performed simulation studies of an exclusive-or classification task with training datasets of 50, 100, and 200 cases (half normal and half abnormal) and an independent testing dataset of 2000 cases. They also compared the methods using a breast ultrasound dataset of 1126 cases. For simulated training datasets of 50 cases, the area under the receiver operating characteristic curve (AUC) was greater (by 0.03) when training with noise injection than when training without any regularization, and the improvement was greater than those from weight decay and early stopping (both of 0.02). For training datasets of 100 cases, noise injection and weight decay yielded similar increases in the AUC (0.02), whereas early stopping produced a smaller increase (0.01). For training datasets of 200 cases, the increases in the AUC were negligibly small for all methods (0.005). For the ultrasound dataset, noise injection had a greater average AUC than ANNs trained without regularization and a slightly greater average AUC than ANNs trained with weight decay. These results indicate that training ANNs with noise injection can reduce overfitting to a greater degree than early stopping and to a similar degree as weight decay.

show abstract

Section: Iie Banns and Weight Decaymentioning

confidence: 99%

Section: Iif Early Stoppingmentioning

confidence: 99%

Noise injection for training artificial neural networks: A comparison with weight decay and early stopping

et al. 2009

View full text Add to dashboard Cite

show abstract

“…The (PF,PD) pairs generated by adjusting the algorithm's threshold form an ROC curve. ROC analysis is a more general way to measure a classifier's performance than numerical indices (Yousef et al 2004). An ROC curve offers a visual of the tradeoff between the classifier's ability to correctly detect faultprone modules (PD) and the number of incorrectly classified fault-free modules (PF).…”

Section: Roc Curvementioning

confidence: 99%

Techniques for evaluating fault prediction models

2008

View full text Add to dashboard Cite

Many statistical techniques have been proposed to predict fault-proneness of program modules in software engineering. Choosing the "best" candidate among many available models involves performance assessment and detailed comparison, but these comparisons are not simple due to the applicability of varying performance measures. Classifying a software module as fault-prone implies the application of some verification activities, thus adding to the development cost. Misclassifying a module as fault free carries the risk of system failure, also associated with cost implications. Methodologies for precise evaluation of fault prediction models should be at the core of empirical software engineering research, but have attracted sporadic attention. In this paper, we overview model evaluation techniques. In addition to many techniques that have been used in software engineering studies before, we introduce and discuss the merits of cost curves. Using the data from a public repository, our study demonstrates the strengths and weaknesses of performance evaluation techniques and points to a conclusion that the selection of the "best" model cannot be made without considering project cost characteristics, which are specific in each development environment.

show abstract

“…Hold-out, cross-validation, and bootstrapping methods were used with the available sample to predict the AUC and other performance measures, which were then compared to the performance of the LR classifier designed on the available sample and applied to the independent test set. Yousef et al 10 investigated the effectiveness of different bootstrap techniques in a Monte Carlo simulation study. Neither of the last two studies systematically investigated the effect of feature space dimensionality, class separability, or the performance of LOO and Fukunaga-Hayes ͑F-H͒ resampling methods.…”

Section: Introductionmentioning

confidence: 99%

Classifier performance prediction for computer‐aided diagnosis using a limited dataset

2008

View full text Add to dashboard Cite

In a practical classifier design problem, the true population is generally unknown and the available sample is finite-sized. A common approach is to use a resampling technique to estimate the performance of the classifier that will be trained with the available sample. We conducted a Monte Carlo simulation study to compare the ability of the different resampling techniques in training the classifier and predicting its performance under the constraint of a finite-sized sample. The true population for the two classes was assumed to be multivariate normal distributions with known covariance matrices. Finite sets of sample vectors were drawn from the population. The true performance of the classifier is defined as the area under the receiver operating characteristic curve ͑AUC͒ when the classifier designed with the specific sample is applied to the true population. We investigated methods based on the Fukunaga-Hayes and the leave-one-out techniques, as well as three different types of bootstrap methods, namely, the ordinary, 0.632, and 0.632+ bootstrap. The Fisher's linear discriminant analysis was used as the classifier. The dimensionality of the feature space was varied from 3 to 15. The sample size n 2 from the positive class was varied between 25 and 60, while the number of cases from the negative class was either equal to n 2 or 3n 2 . Each experiment was performed with an independent dataset randomly drawn from the true population. Using a total of 1000 experiments for each simulation condition, we compared the bias, the variance, and the root-mean-squared error ͑RMSE͒ of the AUC estimated using the different resampling techniques relative to the true AUC ͑obtained from training on a finite dataset and testing on the population͒. Our results indicated that, under the study conditions, there can be a large difference in the RMSE obtained using different resampling methods, especially when the feature space dimensionality is relatively large and the sample size is small. Under this type of conditions, the 0.632 and 0.632+ bootstrap methods have the lowest RMSE, indicating that the difference between the estimated and the true performances obtained using the 0.632 and 0.632+ bootstrap will be statistically smaller than those obtained using the other three resampling methods. Of the three bootstrap methods, the 0.632+ bootstrap provides the lowest bias. Although this investigation is performed under some specific conditions, it reveals important trends for the problem of classifier performance prediction under the constraint of a limited dataset.

show abstract

Comparison of Non-Parametric Methods for Assessing Classifier Performance in Terms of ROC Parameters

Cited by 26 publications

References 6 publications

Noise injection for training artificial neural networks: A comparison with weight decay and early stopping

Noise injection for training artificial neural networks: A comparison with weight decay and early stopping

Techniques for evaluating fault prediction models

Classifier performance prediction for computer‐aided diagnosis using a limited dataset

Contact Info

Product

Resources

About