TMSS: An End-to-End Transformer-Based Multimodal Network for Segmentation and Survival Prediction

Abstract

When oncologists estimate cancer patient survival, they rely on multimodal data. Even though some multimodal deep learning methods have been proposed in the literature, the majority rely on having two or more independent networks that share knowledge at a later stage in the overall model. On the other hand, oncologists do not do this in their analysis but rather fuse the information in their brain from multiple sources such as medical images and patient history. This work proposes a deep learning method that mimics oncologists' analytical behavior when quantifying cancer and estimating patient survival. We propose TMSS, an end-to-end Transformer based Multimodal network for Segmentation and Survival predication that leverages the superiority of transformers that lies in their abilities to handle different modalities. The model was trained and validated for segmentation and prognosis tasks on the training dataset from the HEad & NeCK TumOR segmentation and the outcome prediction in PET/CT images challenge (HECKTOR). We show that the proposed prognostic model significantly outperforms state-of-the-art methods with a concordance index of 0.763 ± 0.14 while achieving a comparable dice score of 0.772 ± 0.030 to a standalone segmentation model. The code is publicly available at https://t.ly/V-_W.

Multitask Learning with Convolutional Neural Networks and Vision Transformers Can Improve Outcome Prediction for Head and Neck Cancer Patients

“…The HECKTOR2021 dataset was previously analyzed using multitask ViT models for the simultaneous segmentation and prediction of PFS by Saeed et al [59]. In contrast to our investigation, they optimized the loss function proposed by Yu et al [25] for outcome modeling and did not investigate the impact of having more than one outcome model.…”

Comprehensive review of Transformer‐based models in neuroscience, neurology, and psychiatry

“…It includes a comprehensive evaluation of disease progression, treatment effectiveness, and patient survival. For example, Saeed et al [136] introduced TMSS, an end-to-end Transformer-based multimodal network that mimics the behavior of oncologists in quantifying cancer and estimating patient survival rates. Nguyen et al [137] developed CLIMAT, an interpretable framework for predicting disease trajectory based on multimodal data.…”

Understanding the brain with attention: A survey of transformers in brain sciences

“…[231] Another work demonstrated that prompt integration into learnable parameters can achieve excellent performance in medical image segmentation. [232] Transformers optimize the embedding and task headers in the specific task via prompt-based learning, which guides the distribution of training data close to the task description. Thus, Transformers pay more attention to guidance, pursuing training efficiency and accuracy.…”

“…A study found that prompt learning can significantly improve the representation learning ability of ViTs because it is proven to integrate domain knowledge effectively [231] . Another work demonstrated that prompt integration into learnable parameters can achieve excellent performance in medical image segmentation [232] …”