Foibles, follies, and fusion: Web-based collaboration for medical image labeling

Abstract

Labels that identify specific anatomical and functional structures within medical images are essential to the characterization of the relationship between structure and function in many scientific and clinical studies. Automated methods that allow for high throughput have not yet been developed for all anatomical targets or validated for exceptional anatomies, and manual labeling remains the gold standard in many cases. However, manual placement of labels within a large image volume such as that obtained using magnetic resonance imaging is exceptionally challenging, resource intensive, and fraught with intra- and inter-rater variability. The use of statistical methods to combine labels produced by multiple raters has grown significantly in popularity, in part, because it is thought that by estimating and accounting for rater reliability estimates of the true labels will be more accurate. This paper demonstrates the performance of a class of these statistical label combination methodologies using real-world data contributed by minimally trained human raters. The consistency of the statistical estimates, the accuracy compared to the individual observations, and the variability of both the estimates and the individual observations with respect to the number of labels are presented. It is demonstrated that statistical fusion successfully combines label information using data from online (Internet-based) collaborations among minimally trained raters. This first successful demonstration of a statistically based approach using minimally trained raters opens numerous possibilities for very large scale efforts in collaboration. Extension and generalization of these technologies for new applications will certainly present fascinating areas for continuing research.

Multi-atlas segmentation of biomedical images: A survey

“…An alternative approach is to use automatic segmentations as atlases, after applying a quality control step. Yet a different strategy is to harness the potential of non-expert segmenters (Bogovic et al, 2013; Bryan et al, 2014), for example, via a crowd-sourcing framework (Landman et al, 2012a; Maier-Hein et al, 2014). Although many biomedical segmentation problems rely on anatomical expertise, it is not clear whether this expertise has to be deployed in the delineation of every single atlas.…”

Formulating Spatially Varying Performance in the Statistical Fusion Framework

“…The remaining 7 volumes were used as training data and were used to compute initial estimates of the performance level parameters (see [20] for details). Both STAPLE and Spatial STAPLE used these initial parameter estimates using a bias value of κ = 1.…”

“…Figures 5 and 6) where the foibles an follies of minimally trained raters are taken into account. For example, the concept of crowd-sourcing the labeling problem using minimally trained raters has gained traction [20, 33]. Through simple manipulation of the seed points and window sizes, Spatial STAPLE provides a valuable resource that could, for instance, be used to explicitly model a rater's learning curve, degradation over time, and general human mistakes that are often present in the labeling process.…”

“…Existing methods to simultaneously estimate rater performance (e.g., STAPLE approaches) have used spatially invariant models (i.e. the probability of error does not change voxel-by-voxel) [10, 14-16, 20]. On the other end of the spectrum, voting based techniques (including global [9-11] and local [9, 12, 13] approaches) ignore the spatial relation between voxels and fuse each voxel independently.…”

Collaborative labeling of malignant glioma

“…Crowd-sourced, collaborative labeling has the potential to exploit the “wisdom of the crowd” and avoid strict requirements for particularly “wise” individuals. The recently presented WebMILL framework promises to bring together individuals (i.e., “millers”) independent of physical location and use statistical fusion to combine their results [1]. This potentially transformative technology presents a new set of challenges: investigators must pose the labeling tasks in a manner accessible to people with little or no background in medical imaging and who cannot be expected to read detailed instructions.…”