MC-ViT: Multi-path cross-scale vision transformer for thymoma histopathology whole slide image typing

Abstract

ObjectivesAccurate histological typing plays an important role in diagnosing thymoma or thymic carcinoma (TC) and predicting the corresponding prognosis. In this paper, we develop and validate a deep learning-based thymoma typing method for hematoxylin & eosin (H&E)-stained whole slide images (WSIs), which provides useful histopathology information from patients to assist doctors for better diagnosing thymoma or TC.MethodsWe propose a multi-path cross-scale vision transformer (MC-ViT), which first uses the cross attentive scale-aware transformer (CAST) to classify the pathological information related to thymoma, and then uses such pathological information priors to assist the WSIs transformer (WT) for thymoma typing. To make full use of the multi-scale (10×, 20×, and 40×) information inherent in a WSI, CAST not only employs parallel multi-path to capture different receptive field features from multi-scale WSI inputs, but also introduces the cross-correlation attention module (CAM) to aggregate multi-scale features to achieve cross-scale spatial information complementarity. After that, WT can effectively convert full-scale WSIs into 1D feature matrices with pathological information labels to improve the efficiency and accuracy of thymoma typing.ResultsWe construct a large-scale thymoma histopathology WSI (THW) dataset and annotate corresponding pathological information and thymoma typing labels. The proposed MC-ViT achieves the Top-1 accuracy of 0.939 and 0.951 in pathological information classification and thymoma typing, respectively. Moreover, the quantitative and statistical experiments on the THW dataset also demonstrate that our pipeline performs favorably against the existing classical convolutional neural networks, vision transformers, and deep learning-based medical image classification methods.ConclusionThis paper demonstrates that comprehensively utilizing the pathological information contained in multi-scale WSIs is feasible for thymoma typing and achieves clinically acceptable performance. Specifically, the proposed MC-ViT can well predict pathological information classes as well as thymoma types, which show the application potential to the diagnosis of thymoma and TC and may assist doctors in improving diagnosis efficiency and accuracy.

A survey of Transformer applications for histopathological image analysis: New developments and future directions

“…For breast cancer histopathological image classification, DCET-Net [ 72 ] proposed a dual-stream convolution-expanded transformer architecture; Breast-Net [ 51 ] explores the ability of ensemble learning techniques using four Swin transformer architectures; HATNet [ 52 ] uses end-to-end vision transformers with a self-attention mechanism; ScoreNet [ 16 ] developed an efficient transformer-based architecture that integrates a coarse-grained global attention framework with a fine-grained local attention mechanism framework; LGVIT [ 73 ] built a local–global ViT model by introducing a new local–global MHSA mechanism and a ghost geed-forward network block into the network; dMIL-transformer [ 53 ] developed a two-stage double max–min multiple-instance learning (MIL) transformer architecture that combines both the spatial and morphological information of the cancer regions. Other than breast cancer classification, transformers have also been applied to other histopathological image cancer classification tasks, such as bone cancer classification (NRCA-FCFL [ 74 ]), brain cancer classification (ViT-WSI [ 17 ], ASI-DBNet [ 54 ], Ding et al [ 55 ]), colorectal cancer classification (MIST [ 75 ], DT-DSMIL [ 56 ]), gastric cancer classification (IMGL-VTNet [ 57 ]), kidney subtype classification (i-ViT [ 59 ], tRNAsformer [ 58 ]), thymoma or thymic carcinoma classification (MC-ViT [ 76 ]), lung cancer classification (GTP [ 46 ], FDTrans [ 60 ]), skin cancer classification (Wang et al [ 45 ]), and thyroid cancer classification (Wang et al [ 77 ], PyT2T-ViT [ 41 ], Wang et al [ 78 ]) using different transformer-based architectures. Furthermore, other transformer models such as Transmil [ 65 ], KAT [ 61 ], ViT-based unsupervised contrastive learning architecture [ 79 ], DecT [ 66 ], StoHisNet [ 80 ], CWC-transformer [ 63 ], LA-MIL [ 44 ], SETMIL [ 81 ], Prompt-MIL [ 67 ], GLAMIL [ 67 ], MaskHIT [ 82 ], HAG-MIL [ 68 ], MEGT [ 47 ], MSPT [ 70 ], and HistPathGPT [ 69 ] have also been evaluated on more than one tissue type, such as liver, prostate, breast, brain, gastric, kidney, lung, colorectal, and so on, for h...…”

“…Figure 9 shows some examples of SOTA transformer architectures developed for histopathological image classification. DT-DSMIL [56]), gastric cancer classification (IMGL-VTNet [57]), kidney subtype classification (i-ViT [59], tRNAsformer [58]), thymoma or thymic carcinoma classification (MC-ViT [76]), lung cancer classification (GTP [46], FDTrans [60]), skin cancer classification (Wang et al [45]), and thyroid cancer classification (Wang et al [77], PyT2T-ViT [41], Wang et al [78]) using different transformer-based architectures. Furthermore, other transformer models such as Transmil [65], KAT [61], ViT-based unsupervised contrastive learning architecture [79], DecT [66], StoHisNet [80], CWC-transformer [63], LA-MIL [44], SETMIL [81], Prompt-MIL [67], GLAMIL [67], MaskHIT [82], HAG-MIL [68], MEGT [47], MSPT [70], and HistPathGPT [69] have also been evaluated on more than one tissue type, such as liver, prostate, breast, brain, gastric, kidney, lung, colorectal, and so on, for histopathological image classification using different transformer approaches.…”