On evaluating brain tissue classifiers without a ground truth

Abstract

In this paper, we present a set of techniques for the evaluation of brain tissue classifiers on a large data set of MR images of the head. Due to the difficulty of establishing a gold standard for this type of data, we focus our attention on methods which do not require a ground truth, but instead rely on a common agreement principle. Three different techniques are presented: the Williams' index, a measure of common agreement; STAPLE, an Expectation Maximization algorithm which simultaneously estimates performance parameters and constructs an estimated reference standard; and Multidimensional Scaling, a visualization technique to explore similarity data. We apply these different evaluation methodologies to a set eleven different segmentation algorithms on forty MR images. We then validate our evaluation pipeline by building a ground truth based on human expert tracings. The evaluations with and without a ground truth are compared. Our findings show that comparing classifiers without a gold standard can provide a lot of interesting information. In particular, outliers can be easily detected, strongly consistent or highly variable techniques can be readily discriminated, and the overall similarity between different techniques can be assessed. On the other hand, we also find that some information present in the expert segmentations is not captured by the automatic classifiers, suggesting that common agreement alone may not be sufficient for a precise performance evaluation of brain tissue classifiers.

A New Supervised Method for Blood Vessel Segmentation in Retinal Images by Using Gray-Level and Moment Invariants-Based Features

“…Unlike other authors [52], [53], who selected their training set by random pixel-sample extraction from available manual segmentations of DRIVE and STARE images, we produced our own training set by hand. 1 As discussed in literature, gold-standard images may contain errors (see Bioux et al [61] for a comprehensive discussion on this issue) due to the considerable difficulty involved by the creation of these handmade images. To reduce the risk of introducing errors in and, therefore, of introducing noise in the NN, we opted for carefully selecting specific training samples covering all possible vessel, background, and noise patterns.…”

Neocortical Gray Matter Volume in First-Episode Schizophrenia and First-Episode Affective Psychosis: A Cross-Sectional and Longitudinal MRI Study

“…The second scan occurred on average 1.5 years later. Brain structures were segmented using a validated tissue segmentation algorithm (32,33). Neocortical gray matter (NCGM), cerebral white matter (CWM), SCSF, and lateral ventricles (LV) were measured separately for each hemisphere.…”

Interactive semiautomatic contour delineation using statistical conditional random fields framework

“…Human delineations of medical images are not a true gold standard but are the most objective solution. 18 We refer to Bouix et al 38 for a comprehensive discussion on ground truth in segmentations.…”