Induction in Neuroscience with Classification: Issues and Solutions

Olivetti, Emanuele; Greiner, Susanne; Avesani, Paolo

doi:10.1007/978-3-642-34713-9_6

Cited by 14 publications

(12 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As noted in [ 8 ], when data are unbalanced with respect to the class-label distribution, the PA (or the misclassification error rate) of a classifier can be a misleading statistic to assess whether the classifier actually discriminated the classes or not. An alternative solution to the issue of evaluating classifiers through the error rate/accuracy is testing the full confusion matrix.…”

Section: Methodsmentioning

confidence: 99%

“…The literature answering the question “did the classifier learn to discriminate the classes?” was recently reviewed in [ 8 ], and a novel approach based on the analysis of the statistical independence between predicted and true class labels was proposed based on the work of [ 2 ]. In this work we adopt a similar approach that we summarise here.…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Statistical independence for the evaluation of classifier-based diagnosis

2014

Self Cite

View full text Add to dashboard Cite

Machine learning techniques are increasingly adopted in computer-aided diagnosis. Evaluation methods for classification results that are based on the study of one or more metrics can be unable to distinguish between cases in which the classifier is discriminating the classes from cases in which it is not. In the binary setting, such circumstances can be encountered when data are unbalanced with respect to the diagnostic groups. Having more healthy controls than pathological subjects, datasets meant for diagnosis frequently show a certain degree of unbalancedness. In this work, we propose to recast the evaluation of classification results as a test of statistical independence between the predicted and the actual diagnostic groups. We address the problem within the Bayesian hypothesis testing framework. Different from the standard metrics, the proposed method is able to handle unbalanced data and takes into account the size of the available data. We show experimental evidence of the efficacy of the approach both on simulated data and on real data about the diagnosis of the Attention Deficit Hyperactivity Disorder (ADHD).

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Statistical independence for the evaluation of classifier-based diagnosis

2014

Self Cite

View full text Add to dashboard Cite

show abstract

“…As noted in Olivetti et al (2012a), when the dataset is unbalanced with respect to the class-label distribution, the accuracy (or the error rate) of a classifier can be a misleading statistic to assess whether the classifier actually discriminated the classes. For example, given a test set of 100 instances where 90 are of class 0 and 10 of class 1, a classifier that incurs in 10 misclassification errors, i.e., the estimated error rate is

\hat{ϵ} = 10 / 100 = 0.1

, could be either highly accurate in discriminating the two classes or completely inaccurate.…”

Section: Methodsmentioning

confidence: 99%

“…We draw from the statistics literature and adopt a recent Bayesian test of independence for contingency tables (Casella and Moreno, 2009; Olivetti et al, 2012a). …”

Section: Introductionmentioning

confidence: 99%

ADHD diagnosis from multiple data sources with batch effects

Olivetti¹,

Greiner²,

Avesani³

2012

Front. Syst. Neurosci.

Self Cite

View full text Add to dashboard Cite

The Attention Deficit Hyperactivity Disorder (ADHD) affects the school-age population and has large social costs. The scientific community is still lacking a pathophysiological model of the disorder and there are no objective biomarkers to support the diagnosis. In 2011 the ADHD-200 Consortium provided a rich, heterogeneous neuroimaging dataset aimed at studying neural correlates of ADHD and to promote the development of systems for automated diagnosis. Concurrently a competition was set up with the goal of addressing the wide range of different types of data for the accurate prediction of the presence of ADHD. Phenotypic information, structural magnetic resonance imaging (MRI) scans and resting state fMRI recordings were provided for nearly 1000 typical and non-typical young individuals. Data were collected by eight different research centers in the consortium. This work is not concerned with the main task of the contest, i.e., achieving a high prediction accuracy on the competition dataset, but we rather address the proper handling of such a heterogeneous dataset when performing classification-based analysis. Our interest lies in the clustered structure of the data causing the so-called batch effects which have strong impact when assessing the performance of classifiers built on the ADHD-200 dataset. We propose a method to eliminate the biases introduced by such batch effects. Its application on the ADHD-200 dataset generates such a significant drop in prediction accuracy that most of the conclusions from a standard analysis had to be revised. In addition we propose to adopt the dissimilarity representation to set up effective representation spaces for the heterogeneous ADHD-200 dataset. Moreover we propose to evaluate the quality of predictions through a recently proposed test of independence in order to cope with the unbalancedness of the dataset.

show abstract

“…When D test is imbalanced, i.e. when n and m considerably differ, the interpretation of accuracy as a measure of discrimination, can problematic [29,30]. An alternative measure that reduces the impact of this problem is balanced accuracy [29] The estimation of performance measure like acc or acc B may have high variability for small N .…”

Section: Classification-based Testmentioning

confidence: 99%

The Kernel Two-Sample Test vs. Brain Decoding

Olivetti

Benozzo

Kia

et al. 2013

2013 International Workshop on Pattern Recognition in Neuroimaging

Self Cite

View full text Add to dashboard Cite

In clinical and neuroscientific studies, systematic differences between two populations of brain networks are investigated in order to characterize mental diseases or processes. Those networks are usually represented as graphs built from neuroimaging data and studied by means of graph analysis methods. The typical machine learning approach to study these brain graphs creates a classifier and tests its ability to discriminate the two populations. In contrast to this approach, in this work we propose to directly test whether two populations of graphs are different or not, by using the kernel two-sample test (KTST), without creating the intermediate classifier. We claim that, in general, the two approaches provides similar results and that the KTST requires much less computation. Additionally, in the regime of low sample size, we claim that the KTST has lower frequency of Type II error than the classification approach. Besides providing algorithmic considerations to support these claims, we show strong evidence through experiments and one simulation.

show abstract

Induction in Neuroscience with Classification: Issues and Solutions

Cited by 14 publications

References 17 publications

Statistical independence for the evaluation of classifier-based diagnosis

Statistical independence for the evaluation of classifier-based diagnosis

ADHD diagnosis from multiple data sources with batch effects

The Kernel Two-Sample Test vs. Brain Decoding

Contact Info

Product

Resources

About