Deep Learning Based on Standard H&amp;E Images of Primary Melanoma Tumors Identifies Patients at Risk for Visceral Recurrence and Death

Kulkarni, Prathamesh M.; Robinson, Eric; Pradhan, Jaya Sarin; Gartrell, Robyn D.; Rohr, Bethany R.; Trager, Megan H.; Geskin, Larisa J.; Kluger, Harriet M.; Wong, Pok Fai; Ács, Balázs; Rizk, Emanuelle M.; Yang, Chen; Mondal, Manas; Moore, Michael R.; Osman, Iman; Phelps, Robert; Horst, Basil A.; Chen, Zhe; Ferringer, Tammie; Rimm, David L.; Wang, Jing; Saenger, Yvonne M.

doi:10.1158/1078-0432.ccr-19-1495

Cited by 90 publications

(64 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In two recent studies, neural network-based analyses of WSIs proved to be an effective tool for prognosticating survival outcomes in patients with melanoma (27,28). Given the success of these and other computer vision models, there is growing interest in whether neural networks can be used to predict response to treatments.…”

Section: Discussionmentioning

confidence: 99%

Using Machine Learning Algorithms to Predict Immunotherapy Response in Patients with Advanced Melanoma

Johannet

Coudray²,

Donnelly

et al. 2021

Clinical Cancer Research

Self Cite

125

View full text Add to dashboard Cite

Purpose: Several biomarkers of response to immune checkpoint inhibitors (ICI) show potential but are not yet scalable to the clinic. We developed a pipeline that integrates deep learning on histology specimens with clinical data to predict ICI response in advanced melanoma. Experimental Design: We used a training cohort from New York University (New York, NY) and a validation cohort from Vanderbilt University (Nashville, TN). We built a multivariable classifier that integrates neural network predictions with clinical data. A ROC curve was generated and the optimal threshold was used to stratify patients as high versus low risk for progression. Kaplan–Meier curves compared progression-free survival (PFS) between the groups. The classifier was validated on two slide scanners (Aperio AT2 and Leica SCN400). Results: The multivariable classifier predicted response with AUC 0.800 on images from the Aperio AT2 and AUC 0.805 on images from the Leica SCN400. The classifier accurately stratified patients into high versus low risk for disease progression. Vanderbilt patients classified as high risk for progression had significantly worse PFS than those classified as low risk (P = 0.02 for the Aperio AT2; P = 0.03 for the Leica SCN400). Conclusions: Histology slides and patients' clinicodemographic characteristics are readily available through standard of care and have the potential to predict ICI treatment outcomes. With prospective validation, we believe our approach has potential for integration into clinical practice.

show abstract

Section: Discussionmentioning

confidence: 99%

Using Machine Learning Algorithms to Predict Immunotherapy Response in Patients with Advanced Melanoma

Johannet

Coudray²,

Donnelly

et al. 2021

Clinical Cancer Research

Self Cite

125

View full text Add to dashboard Cite

show abstract

“…46 Concurrently, disease-specific survival was estimated by DL-based prediction of the development of distant metastatic recurrence in patients with primary melanoma. 49 This is a prime example showing that it is possible to train DL networks on clinical endpoints directly from histology. Moreover, this process could even reveal new morphological biomarkers by highlighting specific structures and regions.…”

Section: Prediction Of Genotype and Gene Expressionmentioning

confidence: 99%

Deep learning in cancer pathology: a new generation of clinical biomarkers

et al. 2020

View full text Add to dashboard Cite

Clinical workflows in oncology rely on predictive and prognostic molecular biomarkers. However, the growing number of these complex biomarkers tends to increase the cost and time for decision-making in routine daily oncology practice; furthermore, biomarkers often require tumour tissue on top of routine diagnostic material. Nevertheless, routinely available tumour tissue contains an abundance of clinically relevant information that is currently not fully exploited. Advances in deep learning (DL), an artificial intelligence (AI) technology, have enabled the extraction of previously hidden information directly from routine histology images of cancer, providing potentially clinically useful information. Here, we outline emerging concepts of how DL can extract biomarkers directly from histology images and summarise studies of basic and advanced image analysis for cancer histology. Basic image analysis tasks include detection, grading and subtyping of tumour tissue in histology images; they are aimed at automating pathology workflows and consequently do not immediately translate into clinical decisions. Exceeding such basic approaches, DL has also been used for advanced image analysis tasks, which have the potential of directly affecting clinical decision-making processes. These advanced approaches include inference of molecular features, prediction of survival and end-to-end prediction of therapy response. Predictions made by such DL systems could simplify and enrich clinical decision-making, but require rigorous external validation in clinical settings.

show abstract

“…Overall, the dataset consisted of 141 WSIs from 56 patients, 28 patients with FCD IIb, and 28 patients with genetically confirmed TSC. H&E stainings were included due to the proven potential of CNNs to extract information not visible to the human observer in H&E slides, thus eliminating the need for more complex and expensive immunostainings.…”

Section: Methodsmentioning

confidence: 99%

Same same but different: A Web‐based deep learning application revealed classifying features for the histopathologic distinction of cortical malformations

et al. 2020

View full text Add to dashboard Cite

Objective The microscopic review of hematoxylin‐eosin–stained images of focal cortical dysplasia type IIb and cortical tuber of tuberous sclerosis complex remains challenging. Both entities are distinct subtypes of human malformations of cortical development that share histopathological features consisting of neuronal dyslamination with dysmorphic neurons and balloon cells. We trained a convolutional neural network (CNN) to classify both entities and visualize the results. Additionally, we propose a new Web‐based deep learning application as proof of concept of how deep learning could enter the pathologic routine. Methods A digital processing pipeline was developed for a series of 56 cases of focal cortical dysplasia type IIb and cortical tuber of tuberous sclerosis complex to obtain 4000 regions of interest and 200 000 subsamples with different zoom and rotation angles to train a neural network. Guided gradient‐weighted class activation maps (Guided Grad‐CAMs) were generated to visualize morphological features used by the CNN to distinguish both entities. Results Our best‐performing network achieved 91% accuracy and 0.88 area under the receiver operating characteristic curve at the tile level for an unseen test set. Novel histopathologic patterns were found through the visualized Guided Grad‐CAMs. These patterns were assembled into a classification score to augment decision‐making in routine histopathology workup. This score was successfully validated by 11 expert neuropathologists and 12 nonexperts, boosting nonexperts to expert level performance. Significance Our newly developed Web application combines the visualization of whole slide images with the possibility of deep learning–aided classification between focal cortical dysplasia IIb and tuberous sclerosis complex. This approach will help to introduce deep learning applications and visualization for the histopathologic diagnosis of rare and difficult‐to‐classify brain lesions.

show abstract

Deep Learning Based on Standard H&E Images of Primary Melanoma Tumors Identifies Patients at Risk for Visceral Recurrence and Death

Cited by 90 publications

References 33 publications

Using Machine Learning Algorithms to Predict Immunotherapy Response in Patients with Advanced Melanoma

Using Machine Learning Algorithms to Predict Immunotherapy Response in Patients with Advanced Melanoma

Deep learning in cancer pathology: a new generation of clinical biomarkers

Same same but different: A Web‐based deep learning application revealed classifying features for the histopathologic distinction of cortical malformations

Contact Info

Product

Resources

About