Joint region and nucleus segmentation for characterization of tumor infiltrating lymphocytes in breast cancer

Abstract

Histologic assessment of stromal tumor infiltrating lymphocytes (sTIL) as a surrogate of the host immune response has been shown to be prognostic and potentially chemo-predictive in triplenegative and HER2-positive breast cancers. The current practice of manual assessment is prone to intra-and inter-observer variability. Furthermore, the interplay of sTILs, tumor cells, other microenvironment mediators, their spatial relationships, quantity, and other image-based features have yet to be determined exhaustively and systemically. Towards analysis of these aspects, we developed a deep learning based method for joint region-level and nucleus-level segmentation and classification of breast cancer H&E tissue whole slide images. Our proposed method simultaneously identifies tumor, fibroblast, and lymphocyte nuclei, along with key histologic region compartments including tumor and stroma. We also show how the resultant segmentation masks can be combined with seeding approaches to yield accurate nucleus classifications. Furthermore, we outline a simple workflow for calibrating computational scores to human scores for consistency. The pipeline identifies key compartments with high accuracy (Dice= overall: 0.78, tumor: 0.83, and fibroblasts: 0.77). ROC AUC for nucleus classification is high at 0.89 (microaverage), 0.89 (lymphocytes), 0.90 (tumor), and 0.78 (fibroblasts). Spearman correlation between computational sTIL and pathologist consensus is high (R=0.73, p<0.001) and is higher than interpathologist correlation (R=0.66, p<0.001). Both manual and computational sTIL scores successfully stratify patients by clinical progression outcomes.

Deep-Learning–Based Characterization of Tumor-Infiltrating Lymphocytes in Breast Cancers From Histopathology Images and Multiomics Data

“…For instance, Amgad et al 23 has developed an effective deep-learning–based method for joint region-level and nucleus-level segmentation of TILs, even though their work was limited by the lack of validation on large-scale image datasets and additional analysis between spatial TIL features and biologic data. Saltz et al 24 have presented global mappings, as well as the spatial organization and molecular correlation of TILs for over 5,000 H&E diagnostic WSIs from The Cancer Genome Atlas (TCGA) dataset, which represented a benchmark for TIL analysis.…”

High expression of MKK3 is associated with worse clinical outcomes in African American breast cancer patients

“…We have previously shown that for this particular dataset, the VGG-16 FCN-8 architecture shows more favorable model convergence and fitting properties than the deeper and more complex DenseNet architecture [29]. Using this particular architecture and number of layers enabled us to leverage the publicly available pre-trained weights, hence improving accuracy [28,29]. The model is trained to classify pixels into one of five classes: tumor (including DCIS), stroma, tumor-infiltrating lymphocytes (including plasma cells and mixed inflammatory infiltrates), necrosis or debris, and others.…”

“…To extract tumor features we used our established standard 16-layer VGG fully-convolutional neural network (VGG16-FCN8) constructed using ImageNet [ 27 ] pre-trained weights as described previously [ 28 ]. We have previously shown that for this particular dataset, the VGG-16 FCN-8 architecture shows more favorable model convergence and fitting properties than the deeper and more complex DenseNet architecture [ 29 ]. Using this particular architecture and number of layers enabled us to leverage the publicly available pre-trained weights, hence improving accuracy [ 28 , 29 ].…”

“…We have previously shown that for this particular dataset, the VGG-16 FCN-8 architecture shows more favorable model convergence and fitting properties than the deeper and more complex DenseNet architecture [ 29 ]. Using this particular architecture and number of layers enabled us to leverage the publicly available pre-trained weights, hence improving accuracy [ 28 , 29 ].…”

Automated Quantification of sTIL Density with H&E-Based Digital Image Analysis Has Prognostic Potential in Triple-Negative Breast Cancers

“…Most studies of computational TILs have employed patch- or object detection-based approaches [ 26 , 27 , 28 , 29 ] with manual region outlining as part of the pipeline [ 30 ]. Some of these also used multiplexed immunofluorescence (mIF) [ 31 ] or immunohistochemistry (IHC) [ 32 , 33 ] to classify cells as lymphocytes.…”