Instance segmentation for whole slide imaging: end-to-end or detect-then-segment

Abstract

Automatic instance segmentation of glomeruli within kidney Whole Slide Imaging (WSI) is essential for clinical research in renal pathology. In computer vision, the end-to-end instance segmentation methods (e.g., Mask-RCNN) have shown their advantages relative to detect-then-segment approaches by performing complementary detection and segmentation tasks simultaneously. As a result, the end-to-end Mask-RCNN approach has been the de facto standard method in recent glomerular segmentation studies, where downsampling and patch-based techniques are used to properly evaluate the high resolution images from WSI (e.g., > 10,000×10,000 pixels on 40×). However, in high resolution WSI, a single glomerulus itself can be more than 1,000×1,000 pixels in original resolution which yields significant information loss when the corresponding features maps are downsampled to the 28×28 resolution via the endto-end Mask-RCNN pipeline. In this paper, we assess if the end-to-end instance segmentation framework is optimal for high-resolution WSI objects by comparing Mask-RCNN with our proposed detect-then-segment framework. Beyond such a comparison, we also comprehensively evaluate the performance of our detect-then-segment pipeline through: 1) two of the most prevalent segmentation backbones (U-Net and DeepLab v3); 2) six different image resolutions (512×512, 256×256, 128×128, 64×64, 32×32, and 28×28); and 3) two different color spaces (RGB and LAB). Our detect-then-segment pipeline, with the DeepLab v3 segmentation framework operating on previously detected glomeruli of 512×512 resolution, achieved a 0.953 dice similarity coefficient (DSC), compared with a 0.902 DSC from the end-to-end Mask-RCNN pipeline. Further, we found that neither RGB nor LAB color spaces yield better performance when compared against each other in the context of a detectthen-segment framework. Detect-then-segment pipeline achieved better segmentation performance compared with End-to-end method. This study provides an extensive quantitative reference for other researchers to select the optimized and most accurate segmentation approach for glomeruli, or other biological objects of similar character, on high-resolution WSI.

Glo-In-One: Holistic Glomerular Detection, Segmentation, and Lesion Characterization with Large-scale Web Image Mining

“…2) Results: From the results (Table . IVb), our pipeline yielded decent DSC values across two backbone methods at different training scenarios, which is comparable to the prior arts [70]. Among the benchmark methods, the DeepLab v3 with 512×512 image resolution is preferred.…”

“…Bouteldja et al investigated the concept of active learning for accurate segmentation accuracy [65] by performing a large number of 72,722 expert-based annotations, while Gadermayr et al has proposed a weakly supervised pipeline for segmenting renal glomeruli [66]. Other method, to achieve instance object level segmentation, has integrated fully convolutional networks with detection methods on WSIs(e.g., Mask-RCNN [67]), but there is still room to improve [68], [69].Recently, Aadarsh et al [70] shows that the "detect-then-segment" two-stage segmentation approach yields more accurate results than the conventional single-stage instance segmentation tactics. Following this study, we propose a detect-classify-segment pipeline to achieve even more accurate glomerular segmentation results.…”

“…The glomerular segmentation function is implemented based on our previously proposed "Detect-then-Segment" method [70] (Fig. 3).…”

“…In segmentation, instead of directly apply segmentation approaches on lower resolution image patches, our "Detect-then-Segment" two-stage design [70] applies DeepLab v3 [74] on the detected high-resolution glomerular patches with minized information loss. Such design has been demonstrated to outperform the prevelent singlestage approaches [70]. The DeepLab v3 [74] is employed as the segmentation backbone.…”

Holistic Fine-grained GGS Characterization: From Detection to Unbalanced Classification

“…We propose a holistic glomeruli detection and classification pipeline by integrating our classifier as a post-processing step for glomeruli detection algorithms, where non-glomerulus patches will be identified and filtered out. Closely related to our proposed pipeline is work by Jha et al [27], where the authors show that for high-resolution medical imaging data, a detect-then-segment pipeline performs better than the de facto standard end-to-end segmentation pipeline such as the Mask-RCNN [28]. Thus, it is promising to adapt such a "two-stage" design, where the inputs of classification on highresolution glomeruli images are cropped from high-resolution WSIs instead of low-resolution feature maps that are used for the detection.…”