Transformer-based biomarker prediction from colorectal cancer histology: A large-scale multicentric study

Wagner, Sophia J.; Reisenbüchler, Daniel; West, Nicholas P.; Niehues, Jan Moritz; Zhu, Jiefu; Foersch, Sebastian; Veldhuizen, Gregory Patrick; Quirke, Philip; Grabsch, Heike I.; van den Brandt, Piet A.; Hutchins, Gordon G.A.; Richman, Susan D.; Yuan, Tanwei; Langer, Rupert; Jenniskens, Josien C.A.; Offermans, Kelly; Mueller, Wolfram; Gray, Richard; Gruber, Stephen B.; Greenson, Joel K.; Rennert, Gad; Bonner, Joseph D.; Schmolze, Daniel; Jonnagaddala, Jitendra; Hawkins, Nicholas J.; Ward, Robyn L.; Morton, Dion; Seymour, Matthew; Magill, Laura; Nowak, Marta; Hay, Jennifer; Koelzer, Viktor H.; Church, David N.; Matek, Christian; Geppert, Carol; Peng, Chaolong; Zhi, Cheng; Ouyang, Xiaoming; James, Jacqueline A.; Loughrey, Maurice B.; Salto-Tellez, Manuel; Brenner, Hermann; Hoffmeister, Michael; Truhn, Daniel; Schnabel, Julia A.; Boxberg, Melanie; Peng, Tingying; Kather, Jakob Nikolas; Church, David; Domingo, Enric; Edwards, Joanne; Glimelius, Bengt; Gogenur, Ismail; Harkin, Andrea; Hay, Jen; Iveson, Timothy; Jaeger, Emma; Kelly, Caroline; Kerr, Rachel; Maka, Noori; Morgan, Hannah; Oien, Karin; Orange, Clare; Palles, Claire; Roxburgh, Campbell; Sansom, Owen; Saunders, Mark; Tomlinson, Ian

doi:10.1016/j.ccell.2023.08.002

Cited by 67 publications

(28 citation statements)

References 57 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…TransMIL for whole-slide image classification was investigated in our recent study 53 and in a study by Wagner et al 54 . For a given cytological image we first split it into multiple 224 × 224 image patches.…”

Section: Methodsmentioning

confidence: 99%

Prediction of tumor origin in cancers of unknown primary origin with cytology-based deep learning

Tian,

Liu,

Wei

et al. 2024

Nat Med

View full text Add to dashboard Cite

Cancer of unknown primary (CUP) site poses diagnostic challenges due to its elusive nature. Many cases of CUP manifest as pleural and peritoneal serous effusions. Leveraging cytological images from 57,220 cases at four tertiary hospitals, we developed a deep-learning method for tumor origin differentiation using cytological histology (TORCH) that can identify malignancy and predict tumor origin in both hydrothorax and ascites. We examined its performance on three internal (n = 12,799) and two external (n = 14,538) testing sets. In both internal and external testing sets, TORCH achieved area under the receiver operating curve values ranging from 0.953 to 0.991 for cancer diagnosis and 0.953 to 0.979 for tumor origin localization. TORCH accurately predicted primary tumor origins, with a top-1 accuracy of 82.6% and top-3 accuracy of 98.9%. Compared with results derived from pathologists, TORCH showed better prediction efficacy (1.677 versus 1.265, P < 0.001), enhancing junior pathologists’ diagnostic scores significantly (1.326 versus 1.101, P < 0.001). Patients with CUP whose initial treatment protocol was concordant with TORCH-predicted origins had better overall survival than those who were administrated discordant treatment (27 versus 17 months, P = 0.006). Our study underscores the potential of TORCH as a valuable ancillary tool in clinical practice, although further validation in randomized trials is warranted.

show abstract

Section: Methodsmentioning

confidence: 99%

Prediction of tumor origin in cancers of unknown primary origin with cytology-based deep learning

Tian,

Liu,

Wei

et al. 2024

Nat Med

View full text Add to dashboard Cite

show abstract

“…The H&E-based one-arm model took as input the bag of 180µm patch-level features of size 3,456 extracted from EsVIT 45 where the number of patches per bag varies. We subsequently adapted the state-of-art WSI classification architectures 13,20 for our distant recurrence prediction task. Given a batch size of one, the time scale was discretized into four intervals based on the quartiles of the distribution of uncensored patients and the negative log likelihood loss were used 46 .…”

Section: Methodsmentioning

confidence: 99%

“…Given a batch size of one, the time scale was discretized into four intervals based on the quartiles of the distribution of uncensored patients and the negative log likelihood loss were used 46 . Our ablation study on H&E-based one-arm model showed that the attention-based multiple instance learning model (AttnMIL) 19 outperformed spatial context-aware architectures including graph attention network 13 and transformers 20 while featuring far lower computational complexity (Supplementary Table 3).…”

Section: Methodsmentioning

confidence: 99%

“…Deep learning (DL) models, including those using digitized hematoxylin-and-eosin-stained (H&E) tumor slides have shown great promise in prediction of molecular alterations 9–11 , cell composition 12 , and prognosis 13–18 , outperforming standard pathologist-based assessment. This is particularly true of the latest generation of self-supervised learning and whole slide image (WSI) prediction DL models which use attention-based networks 19 , graphs 13,17 , or vision transformers 20,21 to provide more granular and interpretable image representation. Additionally, multimodal DL models for prognosis prediction are promising to outperform unimodal approaches that solely rely on the morphological information provided by H&E WSIs 14 .…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

HECTOR: multimodal deep learning predicts recurrence risk in endometrial cancer

Fremond-Volinsky,

Horeweg,

Andani

et al. 2023

Preprint

Self Cite

View full text Add to dashboard Cite

Predicting distant recurrence of endometrial cancer (EC) is crucial for personalized adjuvant treatment. The current gold standard of combined pathological and molecular profiling is costly, hampering implementation. We developed HECTOR (Histopathology-basedEndometrialCancerTailoredOutcomeRisk), a multimodal deep learning prognostic model using hematoxylin-and-eosin-stained whole-slide-images and tumor stage as input, on 1,912 patients from seven EC cohorts including the PORTEC-1/-2/-3 randomized trials. HECTOR demonstrated C-indices in internal (n= 353) and external (n= 151) test sets of 0.788 and 0.816 respectively, outperforming the current gold-standard, and identified patients with markedly different outcomes (10-year distant recurrence-free probabilities of 97.0%, 77.7% and 58.1% for HECTOR low, intermediate and high risk groups). HECTOR also predicted adjuvant chemotherapy benefit better than current methods. Morphological and genomic feature extraction identified correlates of HECTOR risk groups, some with therapeutic potential. HECTOR improves on the current gold-standard and may help delivery of personalized treatment in EC.

show abstract

“…Self-supervised learning (SSL) on the other hand, has gained significantly increasing attention for its capacity to automatically capture image features from unlabeled data[9]. Applications of SSL models have demonstrated superior performance in various cancer classification and survival prediction tasks compared to traditional supervised learning models[14, 15, 16, 17]. Barlow Twins, an SSL model designed to learn non-redundant image features, has several advantages over other SSL learning models (e.g.…”

Section: Introductionmentioning

confidence: 99%

Self-Supervised Learning Reveals Clinically Relevant Histomorphological Patterns for Therapeutic Strategies in Colon Cancer

Liu,

Polack,

Coudray

et al. 2024

Preprint

View full text Add to dashboard Cite

Self-supervised learning (SSL) automates the extraction and interpretation of histopathology features on unannotated hematoxylin-and-eosin-stained whole-slide images (WSIs). We trained an SSL Barlow Twins-encoder on 435 TCGA colon adenocarcinoma WSIs to extract features from small image patches. Leiden community detection then grouped tiles into histomorphological phenotype clusters (HPCs). HPC reproducibility and predictive ability for overall survival was confirmed in an independent clinical trial cohort (N=1213 WSIs). This unbiased atlas resulted in 47 HPCs displaying unique and sharing clinically significant histomorphological traits, highlighting tissue type, quantity, and architecture, especially in the context of tumor stroma. Through in-depth analysis of these HPCs, including immune landscape and gene set enrichment analysis, and association to clinical outcomes, we shed light on the factors influencing survival and responses to treatments like standard adjuvant chemotherapy and experimental therapies. Further exploration of HPCs may unveil new insights and aid decision-making and personalized treatments for colon cancer patients.

show abstract

Transformer-based biomarker prediction from colorectal cancer histology: A large-scale multicentric study

Cited by 67 publications

References 57 publications

Prediction of tumor origin in cancers of unknown primary origin with cytology-based deep learning

Prediction of tumor origin in cancers of unknown primary origin with cytology-based deep learning

HECTOR: multimodal deep learning predicts recurrence risk in endometrial cancer

Self-Supervised Learning Reveals Clinically Relevant Histomorphological Patterns for Therapeutic Strategies in Colon Cancer

Contact Info

Product

Resources

About