Crowdsourcing for translational research: analysis of biomarker expression using cancer microarrays

Abstract

Background:Academic pathology suffers from an acute and growing lack of workforce resource. This especially impacts on translational elements of clinical trials, which can require detailed analysis of thousands of tissue samples. We tested whether crowdsourcing – enlisting help from the public – is a sufficiently accurate method to score such samples.Methods:We developed a novel online interface to train and test lay participants on cancer detection and immunohistochemistry scoring in tissue microarrays. Lay participants initially performed cancer detection on lung cancer images stained for CD8, and we measured how extending a basic tutorial by annotated example images and feedback-based training affected cancer detection accuracy. We then applied this tutorial to additional cancer types and immunohistochemistry markers – bladder/ki67, lung/EGFR, and oesophageal/CD8 – to establish accuracy compared with experts. Using this optimised tutorial, we then tested lay participants' accuracy on immunohistochemistry scoring of lung/EGFR and bladder/p53 samples.Results:We observed that for cancer detection, annotated example images and feedback-based training both improved accuracy compared with a basic tutorial only. Using this optimised tutorial, we demonstrate highly accurate (>0.90 area under curve) detection of cancer in samples stained with nuclear, cytoplasmic and membrane cell markers. We also observed high Spearman correlations between lay participants and experts for immunohistochemistry scoring (0.91 (0.78, 0.96) and 0.97 (0.91, 0.99) for lung/EGFR and bladder/p53 samples, respectively).Conclusions:These results establish crowdsourcing as a promising method to screen large data sets for biomarkers in cancer pathology research across a range of cancers and immunohistochemical stains.

Crowdsourcing of Histological Image Labeling and Object Delineation by Medical Students

“…Crowdsourcing has been used for annotation of high-level objects in microscopic images, but mainly focusing on less complex tasks referred to as microtasks [1]. Successful examples include identification of cancer cells [2], [3], detection of nuclei [4], [5], scoring based on immunohistochemically stained images [2], [3], [6], and detection of Plasmodium falciparum in red blood cells for malaria diagnostics [7]. Other studies focus on the creation of training sets for convolutional neural networks for finding nuclei or mitoses in cancer [8], [9].…”

A Review on the Applications of Crowdsourcing in Human Pathology

“…A 60% drop in academic pathology in the UK between 2000 and 2012,[ 14 ] and a predicted 35% drop in the ratio of pathologists to the population in the US between 2010 and 2030[ 15 ] are evidence that pathology is facing a severe lack of workforce resources. [ 14 ] Automated methods and machine learning are promising methods to automate routine scoring and evaluations[ 16 17 18 19 20 ] However, reducing the workload for pathologists, and development and training of these methods require a large amount of validated datasets which are currently not widely available. [ 14 ] Crowdsourcing, not only may generate the required data for training these algorithms but also can be effectively used to outsource tedious but relatively simple tasks to the crowd, especially in low-resourced areas.…”

“…[ 14 ] Automated methods and machine learning are promising methods to automate routine scoring and evaluations[ 16 17 18 19 20 ] However, reducing the workload for pathologists, and development and training of these methods require a large amount of validated datasets which are currently not widely available. [ 14 ] Crowdsourcing, not only may generate the required data for training these algorithms but also can be effectively used to outsource tedious but relatively simple tasks to the crowd, especially in low-resourced areas. For instance, a pathologist is required to check 1000 of red blood cell (RBC) images for an accurate malaria diagnosis.…”

“…Figure 3 shows the selection flow in a Sankey diagram (Sankey Diagram Generator by Dénes Csala, based on the Sankey plugin for D3 by Mike Bostock; https://sankey.csaladen.es ; 2014). We reviewed the final seven articles,[ 14 21 23 24 25 26 27 ] and extracted their information which is discussed in the next section.…”

“…Among the selected seven articles, three were related to malaria diagnoses,[ 21 23 24 ] and four to cancer diagnoses. [ 14 25 26 27 ] Four topics were covered by the articles: microbiology, immunohistochemistry (IHC), molecular pathology, and in-vivo microscopy. Table 3 summarizes the scope of each article.…”