Topological Feature Extraction and Visualization of Whole Slide Images using Graph Neural Networks

Abstract

Whole-slide images (WSI) are digitized representations of thin sections of stained tissue from various patient sources (biopsy, resection, exfoliation, fluid) and often exceed 100,000 pixels in any given spatial dimension. Deep learning approaches to digital pathology typically extract information from sub-images (patches) and treat the sub-images as independent entities, ignoring contributing information from vital large-scale architectural relationships. Modeling approaches that can capture higher-order dependencies between neighborhoods of tissue patches have demonstrated the potential to improve predictive accuracy while capturing the most essential slide-level information for prognosis, diagnosis and integration with other omics modalities. Here, we review two promising methods for capturing macro and micro architecture of histology images, Graph Neural Networks, which contextualize patch level information from their neighbors through message passing, and Topological Data Analysis, which distills contextual information into its essential components. We introduce a modeling framework, WSI-GTFE that integrates these two approaches in order to identify and quantify key pathogenic information pathways. To demonstrate a simple use case, we utilize these topological methods to develop a tumor invasion score to stage colon cancer.

Mixed Effects Machine Learning Models for Colon Cancer Metastasis Prediction using Spatially Localized Immuno-Oncology Markers

“…Such data modalities require calibration of collection and analysis. For instance, this would require automation of macro-architectural segmentation (intra, inter, away), which can be accomplished through deep learning algorithms 44 . We have yet to assess the potential for bias from semi-subjective/automated selection of ROI within these macroautomated regions.…”

“…For prospective deployment of these technologies, we envision IHC serving as a low-cost alternative and potentially less encumbered by batch effects to rapidly spatially assess markers that have demonstrated utility for identification of colon metastasis. Such data modalities require standardization of collection/analysis processes (e.g., automated segmentation of macroarchitecture (intra, inter, away) with deep learning algorithms, assessment of potential for bias from semi-subjective/automated selection of ROI) 44 . Application of chemical reagents demands further adjustment to ensure meaningful deployment.…”

Graph Neural Networks and Their Current Applications in Bioinformatics

“…Meanwhile, image data can be represented as a graph structure appropriate for the use of GNNs. Therefore, GNNs have an extensive application space in the field of medical imaging, such as image segmentation (Gopinath et al, 2019;Wang et al, 2019b;Tian et al, 2020a,b), abnormal detection (Wu et al, 2019) of MRI images and pathological images, classification (Shi et al, 2019;Zhou et al, 2019;Adnan et al, 2020) and visualization (Levy et al, 2020;Sureka et al, 2020) of histological images, analysis of surgical images , image enhancement , registration (Hansen et al, 2019), retrieval (Zhai et al, 2019), brain connection (Ktena et al, 2017(Ktena et al, , 2018Li X. et al, 2019a;Mirakhorli and Mirakhorli, 2019;Grigis et al, 2020;Zhang and Pierre, 2020;Zhang et al, 2021) and disease prediction (Parisot et al, 2017;Kazi et al, 2018Kazi et al, , 2019aAnirudh and Thiagarajan, 2019;Yang et al, 2019;Stankevičiūtė et al, 2020;Zhang and Pierre, 2020;Zhang et al, 2021), etc.…”

Development of Biologically Interpretable Multimodal Deep Learning Model for Cancer Prognosis Prediction

“…We created a graph convolutional network (GCN) -UniGCN -to predict cancer prognosis from WSI (Figure 1C). GCNs are particularly suited for WSI because convolutions across nodes of a graph, in contrast to Euclidean data, are invariant to permutation and can naturally capture complex macro architectural phenomena in tissue sections operating solely on subarrays of a WSI which exhibit spatial dependence [20]. To our knowledge, this work is one of the first applications of GCNs to WSI to predict cancer survival.…”