Benchmarking artificial intelligence methods for end-to-end computational pathology

Laleh, Narmin Ghaffari; Muti, Hannah Sophie; Loeffler, Chiara Maria Lavinia; Echle, Amelie; Saldanha, Oliver Lester; Mahmood, Faisal; Lu, Ming; Langer, Rupert; Dislich, Bastian; Buelow, Roman D.; Grabsch, Heike; Brenner, Hermann; Chang‐Claude, Jenny; Alwers, Elizabeth; Brinker, Titus J.; Khader, Firas; Truhn, Daniel; Gaisa, Nadine T.; Hoffmeister, Michael; Schulz, Volkmar; Kather, Jakob Nikolas

doi:10.1101/2021.08.09.455633

Cited by 11 publications

(23 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…DeepMed is a straightforward, scalable, and powerful implementation of end-to-end weakly supervised Deep Learning in histology, as demonstrated here. New technologies, such as multiple instance learning 32 and vision transformers 12 , have recently been investigated for such weakly supervised prediction challenges. Although there is limited data on these strategies' realworld performance, some of them may become the de-facto state of the art in the future.…”

Section: Discussionmentioning

confidence: 99%

“…For example, multiple analysis pipelines have been developed between 2018 and 2021 to predict the mutations of oncogenic driver genes from H&E WSI. 3,5,[9][10][11][12] The overall design of these pipelines is largely identical: they load a WSI, tessellate it into tiles, perform data augmentation and/or normalization, train a CNN, deploy the network on tiles from test patients and use an aggregation function to pool the tile-level predictions on a patient level. 9…”

Section: Introduction End-to-end Deep Learning In Computational Pathologymentioning

confidence: 99%

See 1 more Smart Citation

DeepMed: A unified, modular pipeline for end-to-end deep learning in computational pathology

Treeck

Cifci

Laleh

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

The interpretation of digitized histopathology images has been transformed thanks to artificial intelligence (AI). End-to-end AI algorithms can infer high-level features directly from raw image data, extending the capabilities of human experts. In particular, AI can predict tumor subtypes, genetic mutations and gene expression directly from hematoxylin and eosin (H&E) stained pathology slides. However, existing end-to-end AI workflows are poorly standardized and not easily adaptable to new tasks. Here, we introduce DeepMed, a Python library for predicting any high-level attribute directly from histopathological whole slide images alone, or from images coupled with additional meta-data (https://github.com/KatherLab/deepmed). Unlike earlier computational pipelines, DeepMed is highly developer-friendly: its structure is modular and separates preprocessing, training, deployment, statistics, and visualization in such a way that any one of these processes can be altered without affecting the others. Also, DeepMed scales easily from local use on laptop computers to multi-GPU clusters in cloud computing services and therefore can be used for teaching, prototyping and for large-scale applications. Finally, DeepMed is user-friendly and allows researchers to easily test multiple hypotheses in a single dataset (via cross-validation) or in multiple datasets (via external validation). Here, we demonstrate and document DeepMed's abilities to predict molecular alterations, histopathological subtypes and molecular features from routine histopathology images, using a large benchmark dataset which we release publicly. In summary, DeepMed is a fully integrated and broadly applicable end-to-end AI pipeline for the biomedical research community.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Introduction End-to-end Deep Learning In Computational Pathologymentioning

confidence: 99%

DeepMed: A unified, modular pipeline for end-to-end deep learning in computational pathology

Treeck

Cifci

Laleh

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…For prediction of molecular features from image data, we adapted our weakly-supervised prediction pipeline "Histology Image Analysis (HIA)" 9 which was demonstrated to outperform similar approaches for mutation prediction in a recent benchmark study. 37 Briefly, the workflow entails the following steps: As preprocessing step, high resolution WSIs were tessellated into patches of size ሺ512 ൈ 512 ൈ 3ሻ pixels and color-normalized. 38 During this process, blurry patches and patches with no tissue are removed from the data set using canny edge detection in OpenCV.…”

Section: Deep Learning and Swarm Learning Methodsmentioning

confidence: 99%

“…38 During this process, blurry patches and patches with no tissue are removed from the data set using canny edge detection in OpenCV. 37 Subsequently, we used ResNet18 to extract a ሺ512 ൈ 1ሻ feature vector from 150 randomly selected patches for each patient, as previous work showed that 150 patches are sufficient to obtain robust predictions. 9 Feature vectors and patient-wise target labels (BRAF or MSI status) served as input to a fully connected classification network (FCN).…”

Section: Deep Learning and Swarm Learning Methodsmentioning

confidence: 99%

Swarm learning for decentralized artificial intelligence in cancer histopathology

Saldanha

Quirke

West

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Artificial Intelligence (AI) can extract clinically actionable information from medical image data. In cancer histopathology, AI can be used to predict the presence of molecular alterations directly from routine histopathology slides. However, training robust AI systems requires large datasets whose collection faces practical, ethical and legal obstacles. These obstacles could be overcome with swarm learning (SL) where partners jointly train AI models, while avoiding data transfer and monopolistic data governance. Here, for the first time, we demonstrate the successful use of SL in large, multicentric datasets of gigapixel histopathology images comprising over 5000 patients. We show that AI models trained using Swarm Learning can predict BRAF mutational status and microsatellite instability (MSI) directly from hematoxylin and eosin (H&E)-stained pathology slides of colorectal cancer (CRC). We trained AI models on three patient cohorts from Northern Ireland, Germany and the United States of America and validated the prediction performance in two independent datasets from the United Kingdom using SL-based AI models. Our data show that SL enables us to train AI models which outperform most locally trained models and perform on par with models which are centrally trained on the merged datasets. In addition, we show that SL-based AI models are data efficient and maintain a robust performance even if only subsets of local datasets are used for training. In the future, SL can be used to train distributed AI models for any histopathology image analysis tasks, overcoming the need for data transfer and without requiring institutions to give up control of the final AI model.

show abstract

“…To date, only very few studies have investigated the use of ViTs in computational pathology. 23,24 Technical studies have described improved robustness of ViTs to adversarial changes to the input data, but this has not been explored in medical applications. 25–27…”

Section: Introductionmentioning

confidence: 99%

Adversarial attacks and adversarial robustness in computational pathology

Laleh

Truhn

Veldhuizen

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Artificial Intelligence (AI) can support diagnostic workflows in oncology by aiding diagnosis and providing biomarkers. AI applications are therefore expected to evolve from academic prototypes to commercial products in the coming years. However, AI applications are vulnerable to adversarial attacks, such as malicious interference with test data aiming to cause misclassifications. Therefore, it is essential for the use of AI-based diagnostic devices to secure them against such attacks before widespread use. Unfortunately, no resistant systems exist in computational pathology so far. To address this problem, we investigate the susceptibility of convolutional neural networks (CNNs) to multiple types of white- and black-box attacks. We demonstrate that both attacks can easily confuse CNNs in clinically relevant pathology tasks and impair classification performance. Classical adversarially robust training and dual batch normalization (DBN) are possible mitigation strategies but require precise knowledge of the type of attack used in the inference. We demonstrate that vision transformers (ViTs) perform equally well compared to CNNs at baseline and are orders of magnitude more robust to different types of white-box and black-box attacks. At a mechanistic level, we show that this is associated with a more robust latent representation of clinically relevant categories in ViTs compared to CNNs. Our results are in line with previous theoretical studies. We show that ViTs are robust learners in computational pathology. This implies that large-scale rollout of AI models in computational pathology should rely on ViTs rather than CNN-based classifiers to provide inherent protection against adversaries.

show abstract

Benchmarking artificial intelligence methods for end-to-end computational pathology

Cited by 11 publications

References 45 publications

DeepMed: A unified, modular pipeline for end-to-end deep learning in computational pathology

DeepMed: A unified, modular pipeline for end-to-end deep learning in computational pathology

Swarm learning for decentralized artificial intelligence in cancer histopathology

Adversarial attacks and adversarial robustness in computational pathology

Contact Info

Product

Resources

About