Structured crowdsourcing enables convolutional segmentation of histology images

Abstract

Motivation While deep-learning algorithms have demonstrated outstanding performance in semantic image segmentation tasks, large annotation datasets are needed to create accurate models. Annotation of histology images is challenging due to the effort and experience required to carefully delineate tissue structures, and difficulties related to sharing and markup of whole-slide images. Results We recruited 25 participants, ranging in experience from senior pathologists to medical students, to delineate tissue regions in 151 breast cancer slides using the Digital Slide Archive. Inter-participant discordance was systematically evaluated, revealing low discordance for tumor and stroma, and higher discordance for more subjectively defined or rare tissue classes. Feedback provided by senior participants enabled the generation and curation of 20 000+ annotated tissue regions. Fully convolutional networks trained using these annotations were highly accurate (mean AUC=0.945), and the scale of annotation data provided notable improvements in image classification accuracy. Availability and Implementation Dataset is freely available at: https://goo.gl/cNM4EL. Supplementary information Supplementary data are available at Bioinformatics online.

Report on computational assessment of Tumor Infiltrating Lymphocytes from the International Immuno-Oncology Biomarker Working Group

“…A hybrid technique called "internal-external" (cross-) validation may be appropriate when multi-institutional data sets (like the TCGA and METABRIC) are available, where training is performed on some hospitals/institutions and validation is performed on others. This was recommended by Steyerberg and Harrell and used in some computational pathology studies 16,[73][74][75] .…”

“…AV therefore relies on the presence of quality "ground truth" annotations. Unfortunately, there is a lack of open-access, large-scale, multi-institutional histology segmentation and/or TIL classification data sets, with few exceptions 16,24,76,77 . To help address this, a group of scientists, including the US FDA Center for Devices and Radiological Health (CDRH) and the TIL-WG, is collaborating to crowdsource pathologists and collect images and pathologist annotations that can be qualified by the FDA/CDRH medical device development tool program (MDDT).…”

“…Algorithms that learn patterns from labeled data, based on "deep learning" neural networks, have obtained promising results in many challenging problems. Their success has translated well to digital pathology, where they have demonstrated outstanding performance in tasks like mitosis detection, identification of metastases in lymph node sections, tissue segmentation, prognostication, and computational TILs assessment (CTA) [14][15][16][17] . 'Traditional' computational analysis of histology focuses on complex image analysis routines, that typically require extraction of handcrafted features and that often do not generalize well across data sets 18,19 .…”

Utilizing Automated Breast Cancer Detection to Identify Spatial Distributions of Tumor-Infiltrating Lymphocytes in Invasive Breast Cancer

“…In previous work, several research groups carried out image analyses focused on detection of metastatic breast cancer [38][39][40] and mitosis [41][42][43] using highly curated but relatively small datasets from algorithm evaluation challenges [24][25][26][27] 44 proposed a fully convolutional framework for semantic segmentation of histology images via structured crowdsourcing. This was the first work using crowdsourcing in pathology task which involved a total of 25 participants at different expertise levels from medical students to expert pathologists to generate training data for a deep learning algorithm.…”

A Generic Neural Network Approach to Infer Segmenting Classifiers for Disease-Associated Regions in Medical Images

“…As a recent and prominent example, the Gland Segmentation in Colon Histology Images challenge [19] provided a significant number of high-quality reference segmentations performed by a pathologist as ground truth and training data. More recently, the authors of [9] and [2] introduced crowd-based bioimage annotation systems, while the authors in [1] use an image annotation tool to obtain ground truth annotations for different components of lung tissue in 388 sample spots.…”