3D multiplexed tissue imaging reconstruction and optimized region-of-interest (ROI) selection through deep learning model of channels embedding

Abstract

Tissue-based sampling and diagnosis are defined as the extraction of information from certain limited spaces and its diagnostic significance of a certain object. Pathologists deal with issues related to tumor heterogeneity since analyzing a single sample does not necessarily capture a representative depiction of cancer, and a tissue biopsy usually only presents a small fraction of the tumor. Many multiplex tissue imaging platforms (MTIs) make the assumption that tissue microarrays (TMAs) containing small core samples of 2-dimensional (2D) tissue sections are a good approximation of bulk tumors although tumors are not 2D. However, emerging whole slide imaging (WSI) or 3D tumor atlases that employ MTIs like cyclic immunofluorescence (CyCIF) strongly challenge this assumption. In spite of the additional insight gathered by measuring the tumor microenvironment in WSI or 3D, it can be prohibitively expensive and time-consuming to process tens or hundreds of tissue sections with CyCIF. Even when resources are not limited, the criteria for region-of-interest (ROI) selection in tissues for downstream analysis remain largely qualitative and subjective as stratified sampling requires the knowledge of objects and evaluates their features. Despite the fact TMAs fail to adequately approximate whole tissue features, a theoretical subsampling of tissue exists that can best represent the tumor in the whole slide image. To address these challenges, we propose deep learning approaches to learn multi-modal image translation tasks from two aspects: 1) generative modeling approach to reconstruct 3D CyCIF representation and 2) co-embedding CyCIF image and Hematoxylin and Eosin (H&E) section to learn multi-modal mappings by a cross-domain translation for minimum representative ROI selection. We demonstrate that generative modeling enables a 3D virtual CyCIF reconstruction of a colorectal cancer specimen given a small subset of the imaging data at training time. By co-embedding histology and MTI features, we propose a simple convex optimization for objective ROI selection. We demonstrate the potential application of ROI selection and the efficiency of its performance with respect to cellular heterogeneity.

HIPI: Spatially Resolved Multiplexed Protein Expression Inferred from H&E WSIs

“…With this partition, we make sure that tiles used for training and testing come from different areas of the tissue, avoiding memorization and eliminating spatial effects on the results. This is in contrast to previous models that either evaluated on adjacent slices from the same tissue or randomly split regions of the same tissue into training and test sets [15, 16, 17, 20]. Overall, the training set had 1,351,680 tiles, the validation set had 388,783 tiles and the test set had 379,142 tiles.…”

“…Therefore, these models were evaluated only on slices coming from the same tissue used for training but were mostly unsuccessful generalizing to new tissue samples [15, 16, 17]. A very recent article proposed a virtual staining model to map H&E into virtual CyCIF stains [19], but only evaluated on slides coming from the same tissue.…”

COEXIST: Coordinated single-cell integration of serial multiplexed tissue images

“…These types of approaches either require exceptionally thin sections, do not incorporate the full molecular profiles of the cells, or do not perform unified single-cell level analysis 8,19,21,22 . Furthermore, our previous work reported natural variations between serial sections in the image-to-image translation task, especially when using adjacent H&E sections to infer immunofluorescence intensity 18,20 . Recent platforms such as RareCyte Orion 16 or studies 23 are prioritizing multimodal assays such as CyCIF and hematoxylin and eosin (H&E) or high-plex protein combined with whole transcriptome sequencing in the same section to address this problem.…”