Deep learning methods are widely applied in digital pathology to address clinical challenges such as prognosis and diagnosis. As one of the most recent applications, deep models have also been used to extract molecular features from whole slide images. Although molecular tests carry rich information, they are often expensive, time-consuming, and require additional tissue to sample. In this paper, we propose tRNAsformer, an attention-based topology that can learn both to predict the bulk RNA-seq from an image and represent the whole slide image of a glass slide simultaneously. The tRNAsformer uses multiple instance learning to solve a weakly supervised problem while the pixel-level annotation is not available for an image. We conducted several experiments and achieved better performance and faster convergence in comparison to the state-of-the-art algorithms. The proposed tRNAsformer can assist as a computational pathology tool to facilitate a new generation of search and classification methods by combining the tissue morphology and the molecular fingerprint of the biopsy samples.

Nowadays, the availability of different types of biomedical digital data offers many opportunities to investigate the relationships between the different modalities and thus develop a more comprehensive understanding of complex diseases such as cancer. In this paper, we propose a multi-modal model, called deep modality association learning (DMAL), that maps immune cell sequencing patterns to morphological tissue features of whole slide imageds (WSIs) in an embedding space. Useful information is extracted from T-cell receptor (TCR) sequences to guide the training process. DMAL maps the TCR features to the morphology features in histopathology images, which in turn enables the model to learn the association features between the two modalities. The discrimination power of the WSI-TCR association features has been assessed by classifying samples with different cancer subtypes. The conducted experiments have shown that DMAL generates more discriminative features compared to features obtained from single-modal data. In addition, DMAL has been utilized to predict TCR information from histopathology image representations without the need to have the actual TCR sequencing data.