Deep learning-based cross-classifications reveal conserved spatial behaviors within tumor histological images

Abstract

Histopathological images are a rich but incompletely explored data type for studying cancer. Manual inspection is time consuming, making it challenging to use for image data mining. Here we show that convolutional neural networks (CNNs) can be systematically applied across cancer types, enabling comparisons to reveal shared spatial behaviors. We develop CNN architectures to analyze 27,815 hematoxylin and eosin scanned images from The Cancer Genome Atlas for tumor/normal, cancer subtype, and mutation classification. Our CNNs are able to classify TCGA pathologist-annotated tumor/normal status of whole slide images (WSIs) in 19 cancer types with consistently high AUCs (0.995 ± 0.008), as well as subtypes with lower but significant accuracy (AUC 0.87 ± 0.1). Remarkably, tumor/normal CNNs trained on one tissue are effective in others (AUC 0.88 ± 0.11), with classifier relationships also recapitulating known adenocarcinoma, carcinoma, and developmental biology. Moreover, classifier comparisons reveal intra-slide spatial similarities, with an average tile-level correlation of 0.45 ± 0.16 between classifier pairs. Breast cancers, bladder cancers, and uterine cancers have spatial patterns that are particularly easy to detect, suggesting these cancers can be canonical types for image analysis. Patterns for TP53 mutations can also be detected, with WSI self- and cross-tissue AUCs ranging from 0.65-0.80. Finally, we comparatively evaluate CNNs on 170 breast and colon cancer images with pathologist-annotated nuclei, finding that both cellular and intercellular regions contribute to CNN accuracy. These results demonstrate the power of CNNs not only for histopathological classification, but also for cross-comparisons to reveal conserved spatial behaviors across tumors.

Deep Learning of Histopathological Features for the Prediction of Tumour Molecular Genetics

“…In addition to applying deep learning to a single cancer type, researchers have shown that a single neural network may be applied across different tissues. Noorbakhsh et al trained a neural network to predict the mutation status of TP53 using patients with a single cancer type, and showed that the model could generalise to other tissues with comparable results to self-cohort models [44]. For example, a model trained on breast cancer data achieved AUCs of 0.72, 0.71 and 0.67 on lung adenocarcinoma, stomach adenocarcinoma and bladder urothelial carcinoma test sets, respectively, while also achieving similar performance on the lung adenocarcinoma cases to Coudray et al [34].…”

Deep learning predicts chromosomal instability from histopathology images

“…Javad et al. systematically used CNNs on 23 cancer types for tasks including tumor versus normal and cancer subtype classifications as well as predicting the presence of TP53 mutations ( Noorbakhsh, et al., 2020 ).…”

Deep learning trained on H&E tumor ROIs predicts HER2 status and Trastuzumab treatment response in HER2+ breast cancer

“…Representative examples of CNN-based models for pathology applications include tumor/benign classification (23)(24)(25)(26)(27), predicting mutations in key genes (24,27,28), cancer subtype classification and morphology analysis (23,30), and treatment outcome prediction (31,32). These models have shown impressive performance, demonstrating that subtle molecular features of cancer may be discernible from H&E images.…”

“…In recent years, there has been a growth in machine learning approaches, especially deep learning, in the field of pathology (19). These typically utilize Convolutional Neural Network (CNN) architectures, such as AlexNet (20), GoogleNet (21), or ResNet (22), etc., pre-trained on generic images, and then fine-tune them by re-training the last layers for a specific classification task. This approach is typically referred to as "transfer-learning".…”