Spatial Transcriptomics Prediction from Histology jointly through Transformer and Graph Neural Networks

Abstract

The rapid development of spatial transcriptomics allows for the measurement of RNA abundance at a high spatial resolution, making it possible to simultaneously profile gene expression, spatial locations, and the corresponding hematoxylin and eosin-stained histology images. Since histology images are relatively easy and cheap to obtain, it is promising to leverage histology images for predicting gene expression. Though several methods have been devised to predict gene expression using histology images, they don’t simultaneously include the 2D vision features and the spatial dependency, limiting their performances. Here, we have developed Hist2ST, a deep learning-based model using histology images to predict RNA-seq expression. At each sequenced spot, the corresponding histology image is cropped into an image patch, from which 2D vision features are learned through convolutional operations. Meanwhile, the spatial relations with the whole image and neighbored patches are captured through Transformer and graph neural network modules, respectively. These learned features are then used to predict the gene expression by following the zero-inflated negative binomial (ZINB) distribution. To alleviate the impact by the small spatial transcriptomics data, a self-distillation mechanism is employed for efficient learning of the model. Hist2ST was tested on the HER2-positive breast cancer and the cutaneous squamous cell carcinoma datasets, and shown to outperform existing methods in terms of both gene expression prediction and following spatial region identification. Further pathway analyses indicated that our model could reserve biological information. Thus, Hist2ST enables generating spatial transcriptomics data from histology images for elucidating molecular signatures of tissues.

Deep spatial-omics to aid personalization of precision medicine in metastatic recurrent Head & Neck Cancers

“…We envision that spatial RNA/protein analysis can be adopted in the clinical settings, as whole genome sequencing is now a routine test requested by clinicians. Importantly, the cost of these technologies and high technical requirements can already be drastically reduced by the implementation of artificial intelligence (AI) models capable to predict in situ gene expression inferred from fast/low-cost H&E images using curated disease Spatial-omics training data sets 36,37 . Thus, after an initial investment dedicated to create standardized disease-specific AI training material, the spatial data of each patient's tumor biopsy can be obtained (experimentally or AIinferred) and contrasted against spatial databases of the disease to help with different steps along each patient's journey: (i) aid in the annotation of the tumor, (ii) stratify patients based on disease risk progression to personalize surveillance plans; and (iii) to generate a list based on a patient's own disease features which informs oncologists of targets with quantifiable likelihood to have an impact on the disease.…”

Deciphering Spatial Domains by Integrating Histopathological Image and Transcriptomics via Contrastive Learning

“…For the dataset generated by 10x Visium, we used PCA to reduce the dimension of gene expression to 300 as recommended in ref [29,31]. For other datasets, we followed ref [13] to select the top 1000 highly variable genes; For a given spot, its counts were divided by the total counts for the given spot and multiplied by the scale factor of 1,000,000. This was then natural-log transformed via log (1+𝑥𝑥), where 𝑥𝑥 was the normalized count.…”

“…Even though these structure-based methods achieve decent performance on some datasets by depending on the spatial structure and gene expression, their performance may be influenced when the gene expressions contain a mass of noises due to the defect of the current sequencing techniques [25, 26]. Previous methods show that histopathological images can be used for predicting gene expression[13, 27], indicating images also contain abundant information. Thus, it is promising to conquer noises in gene expression by introducing image features as complementary information for gene expression profiles.…”

“…In fact, spatial domain identification could be resolved by applying clustering methods [9][10][11][12]. Nevertheless, conventional clustering methods are meeting grand challenges since they do not incorporate the unique properties of spatial transcriptomics or they are difficult to deal with the sparsity and high dimensional features of gene expression data [13,14].…”