Predicting the Visual Attention of Pathologists Evaluating Whole Slide Images of Cancer

Chakraborty, Souradeep; Gupta, Rajarsi; Ma, Ke; Govind, D.; Sarder, Pinaki; Choi, Won‐Tak; Mahmud, Waqas; Yee, Eric U.; Allard, Felicia D.; Knudsen, Beatrice S.; Zelinsky, Gregory J.; Saltz, Joel; Samaras, Dimitris

doi:10.1007/978-3-031-16961-8_2

Cited by 2 publications

(1 citation statement)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The past two years have seen a surge in popularity of transformer modeling for common computational pathology tasks such as WSI segmentation [40,43,44] and histology image classification [41,[45][46][47]. Transformers have also been used for pathologist-level question-answering from histological imaging [39], predicting pathologists' visual attention [48], and for pathology text mining [49]. Nearly all applications of transformer-based approaches to whole slide imaging implement vision transformers (ViT), including recent works that combine CNNs and transformers [45].…”

Section: Introductionmentioning

confidence: 99%

Spatially Aware Transformer Networks for Contextual Prediction of Diabetic Nephropathy Progression from Whole Slide Images

Shickel

Lucarelli

Rao

et al. 2023

Preprint

Self Cite

View full text Add to dashboard Cite

Diabetic nephropathy (DN) in the context of type 2 diabetes is the leading cause of end-stage renal disease (ESRD) in the United States. DN is graded based on glomerular morphology and has a spatially heterogeneous presentation in kidney biopsies that complicates pathologists predictions of disease progression. Artificial intelligence and deep learning methods for pathology have shown promise for quantitative pathological evaluation and clinical trajectory estimation; but, they often fail to capture large-scale spatial anatomy and relationships found in whole slide images (WSIs). In this study, we present a transformer-based, multi-stage ESRD prediction framework built upon nonlinear dimensionality reduction, relative Euclidean pixel distance embeddings between every pair of observable glomeruli, and a corresponding spatial self-attention mechanism for a robust contextual representation. We developed a deep transformer network for encoding WSI and predicting future ESRD using a dataset of 56 kidney biopsy WSIs from DN patients at Seoul National University Hospital. Using a leave-one-out cross-validation scheme, our modified transformer framework outperformed RNNs, XGBoost, and logistic regression baseline models, and resulted in an area under the receiver operating characteristic curve (AUC) of 0.97 (95% CI: 0.90-1.00) for predicting two-year ESRD, compared with an AUC of 0.86 (95% CI: 0.66-0.99) without our relative distance embedding, and an AUC of 0.76 (95% CI: 0.59-0.92) without a denoising autoencoder module. While the variability and generalizability induced by smaller sample sizes are challenging, our distance-based embedding approach and overfitting mitigation techniques yielded results that suggest opportunities for future spatially aware WSI research using limited pathology datasets.

show abstract