Deep learning-based image analysis methods for brightfield-acquired multiplex immunohistochemistry images

Abstract

Background Multiplex immunohistochemistry (mIHC) permits the labeling of six or more distinct cell types within a single histologic tissue section. The classification of each cell type requires detection of uniquely colored chromogens localized to cells expressing biomarkers of interest. The most comprehensive and reproducible method to evaluate such slides is to employ digital pathology and image analysis pipelines to whole-slide images (WSIs). Our suite of deep learning tools quantitatively evaluates the expression of six biomarkers in mIHC WSIs. These methods address the current lack of readily available methods to evaluate more than four biomarkers and circumvent the need for specialized instrumentation to spectrally separate different colors. The use case application for our methods is a study that investigates tumor immune interactions in pancreatic ductal adenocarcinoma (PDAC) with a customized mIHC panel. Methods Six different colored chromogens were utilized to label T-cells (CD3, CD4, CD8), B-cells (CD20), macrophages (CD16), and tumor cells (K17) in formalin-fixed paraffin-embedded (FFPE) PDAC tissue sections. We leveraged pathologist annotations to develop complementary deep learning-based methods: (1) ColorAE is a deep autoencoder which segments stained objects based on color; (2) U-Net is a convolutional neural network (CNN) trained to segment cells based on color, texture and shape; and (3) ensemble methods that employ both ColorAE and U-Net, collectively referred to as ColorAE:U-Net. We assessed the performance of our methods using: structural similarity and DICE score to evaluate segmentation results of ColorAE against traditional color deconvolution; F1 score, sensitivity, positive predictive value, and DICE score to evaluate the predictions from ColorAE, U-Net, and ColorAE:U-Net ensemble methods against pathologist-generated ground truth. We then used prediction results for spatial analysis (nearest neighbor). Results We observed that (1) the performance of ColorAE is comparable to traditional color deconvolution for single-stain IHC images (note: traditional color deconvolution cannot be used for mIHC); (2) ColorAE and U-Net are complementary methods that detect six different classes of cells with comparable performance; (3) combinations of ColorAE and U-Net in ensemble methods outperform ColorAE and U-Net alone; and (4) ColorAE:U-Net ensemble methods can be employed for detailed analysis of the tumor microenvironment (TME). Summary We developed a suite of scalable deep learning methods to analyze 6 distinctly labeled cell populations in mIHC WSIs. We evaluated our methods and found that they reliably detected and classified cells in the PDAC tumor microenvironment. We also utilized the ColorAE:U-Net ensemble method to analyze 3 mIHC WSIs with nearest neighbor spatial analysis. We demonstrate a proof of concept that these methods can be employed to quantitatively describe the spatial distribution of immune cells within the tumor microenvironment. These complementary deep learning methods are readily deployable for use in clinical research studies.

Deep learning in cancer pathology: a new generation of clinical biomarkers

“…Therefore, H&E slides are routinely available for almost every cancer patient, making them an easy-to-obtain, information-rich data source for the assessment by DL methods, also explaining the focus of previous studies on these types of images. Nevertheless, DL is a tool with applicability in different types of histological stains, such as immunohistochemistry (IHC) 21 or periodic acid-Schiff. 22 …”

Artificial Intelligence in Cancer Research and Precision Medicine

“…In another work, Bejnordi and colleagues trained and tested a DNN (VGG-Net architecture) on histopathology images from breast biopsies of 882 patients to distinguish benign from malignant tissues and classify normal versus tumor-associated stroma with an accuracy of 92% (59). Recently, Fassler and colleagues leveraged histopathology images obtained from multiplex IHC of pancreatic cancer tissue and applied a DNN comprised of an autoencoder (ColorAE) together with a U-Net CNN (60). Cell segmentation and classification performance ranged from 0.40 to 0.84 (expressed as F1 score, an alternative to AUC).…”

State of machine and deep learning in histopathological applications in digestive diseases

“…Multiplex IHC (mIHC) involves concurrent histological staining of 6 cell markers or more, and Abousamra et al [ 56 ] developed an autoencoder-based color deconvolution algorithm to segment these different stains within a WSI. In a follow-up study, Fassler et al [ 57 ] utilized this algorithm on mIHC-stained pancreatic ductal adenocarcinoma (PDAC) WSIs to detect and perform spatial analyses on the cell types. Results indicated that CD16+ myeloid cells dominated the immune microenvironment and on average were of closer distance to tumor cells than CD3+, CD4+, CD8+, or CD20+ lymphocyte populations.…”

“…Results indicated that CD16+ myeloid cells dominated the immune microenvironment and on average were of closer distance to tumor cells than CD3+, CD4+, CD8+, or CD20+ lymphocyte populations. In contrast to the study by Awan et al [ 55 ] that used segmentation outputs to inform a clinical task, Fassler et al [ 57 ] targeted a research application. A pipeline to detect all cell types from mIHC-stained WSIs, quantify, and perform special statistics would serve a wide audience of basic and translational researchers, and, in elevating analytical capacities, may stimulate research output.…”