Dense, high-resolution mapping of cells and tissues from pathology images for the interpretable prediction of molecular phenotypes in cancer

Abstract

While computational methods have made substantial progress in improving the accuracy and throughput of pathology workflows for diagnostic, prognostic, and genomic prediction, lack of interpretability remains a significant barrier to clinical integration. In this study, we present a novel approach for predicting clinically-relevant molecular phenotypes from histopathology whole-slide images (WSIs) using human-interpretable image features (HIFs). Our method leverages >1.6 million annotations from board-certified pathologists across >5,700 WSIs to train deep learning models for high-resolution tissue classification and cell detection across entire WSIs in five cancer types. Combining cell- and tissue-type models enables computation of 607 HIFs that comprehensively capture specific and biologically-relevant characteristics of multiple tumors. We demonstrate that these HIFs correlate with well-known markers of the tumor microenvironment (TME) and can predict diverse molecular signatures, including immune checkpoint protein expression and homologous recombination deficiency (HRD). Our HIF-based approach provides a novel, quantitative, and interpretable window into the composition and spatial architecture of the TME.

Visual Analytics: A Method to Explore Natural Histories of Oral Epithelial Dysplasia

“…Computer vision algorithms can identify cancerous nodules from medical imaging with accuracy sometimes exceeding human experts [ 1 ]. Recent preliminary research has also made headway in making these algorithms more interpretable for clinicians [ 2 ]. However, state-of-the-art algorithms such as these are applied to narrow and highly specific tasks and require large volumes of highly constrained, well-defined data while relying on a number of assumptions about the statistical properties of these data [ 3 ] ( Figure 1A ).…”

Computational pathology: A survey review and the way forward

“… H&E, IHC 6 402 TMAs N/A Slide S 4 I TCGA-Courtiol 87 TCGA H&E 56 Patients, 56 WSIs N/A Patient, Slide S 3 I BreCaHAD 74 link H&E 170 ROIs 40× ROI S 2 U TCGA-Hegde 702 TCGA H&E 60 WSIs 10× ROI S 10 U TCGA-Diao 703 TCGA H&E 2 917 WSIs 20×, 40× ROI, Pixel S 4, 6 I TCGA-Levine 668 TCGA H&E 668 WSIs N/A ROI S 5 U TCGA@Focus 196 Link H&E 1K WSIs, 14 371 Patches N/A Patch S 2 I TCGA-Shen 704 TCGA H&E 1 063 WSIs 20× Patch S 3 U TCGA-Lerousseau 444 TCGA H&E 6 481 WSIs 20× Pixel S 3 U TCGA-Sch...…”

Artificial Intelligence and Cellular Segmentation in Tissue Microscopy Images

“…If a network that segments cancerous tissue is combined with a network that segments cells, the resulting segmentations can be mined for human-interpretable features of the spatial distribution of cells, such as lymphocyte density near the cancer boundary and relative density of immune cells and cancer cells within the tumor. 39 The method of combining deep automated cell and lesion detection with humanengineered computer-visualization features provides highthroughput computation of interpretable features that previously would have required costly and time-intensive manual annotation of images.…”