A Survey of Crowdsourcing in Medical Image Analysis

Abstract

Rapid advances in image processing capabilities have been seen across many domains, fostered by the application of machine learning algorithms to "big-data". However, within the realm of medical image analysis, advances have been curtailed, in part, due to the limited availability of large-scale, well-annotated datasets. One of the main reasons for this is the high cost often associated with producing large amounts of high-quality meta-data. Recently, there has been growing interest in the application of crowdsourcing for this purpose; a technique that has proven effective for creating large-scale datasets across a range of disciplines, from computer vision to astrophysics. Despite the growing popularity of this approach, there has not yet been a comprehensive literature review to provide guidance to researchers considering using crowdsourcing methodologies in their own medical imaging analysis. In this survey, we review studies applying crowdsourcing to the analysis of medical images, published prior to July 2018 . We identify common approaches, challenges and considerations, providing guidance of utility to researchers adopting this approach. Finally, we discuss future opportunities for development within this emerging domain.

Artificial intelligence in glioma imaging: challenges and advances

“…The aforementioned ImageNet dataset [29] has been annotated using a crowd-sourcing platform (Amazon Mechanical Turks). Although there may be few medical experts within the labelling group, crowd-sourcing has been shown to be effective in creating large quantities of annotated data; and, it is faster and cheaper than annotation by medical experts [65]. In neuroscience research, crowdsourcing and gamification have helped neuroscientists to explore brain networks by identifying neurons and their synaptic connections [66,67].…”

Learning to Segment from Scribbles using Multi-scale Adversarial Attention Gates

“…Here we investigate the possibility to train our model using multiple scribbles per training image. This scenario simulates crowdsourcing applications, which have shown to be useful for annotating rare classes or to exploit various levels of expertise in annotators [8], [47]. Here, we mimic the availability of scribble annotations collected by three different "sources", using: i) expert-made scribbles; ii) scribbles approximated by skeletonization of the segmentation masks; iii) scribbles approximated by a random walk in the masks (see Section IV-A for a description of ii) and iii)).…”

“…Thus, we consider multiple times pixels that are labeled across annotators, while considering 'once' pixels labeled only from one annotator. Other ways of combining annotations are also possible (e.g, considering the union of the scribbles, or weighting differently each annotator [47]), but they are out of the scope of this manuscript.…”

Label noise in segmentation networks : mitigation must deal with bias

“…True labels can be estimated from multiple imperfect reference labels with expectation minimization (EM) [10,11,12,13]. Creating multiple segmentation annotations is typically too expensive but lower quality results can be obtained with crowdsourcing [15]. Indeed, it has been demonstrated that the most efficient labeling strategy is to collect one high quality label per example for many examples and then estimate the true labels with model-bootstrapped EM [14].…”