Softmax-Driven Active Shape Model for Segmenting Crowded Objects in Digital Pathology Images

Abstract

Automated segmentation of histological structures in microscopy images is a crucial step in computer-aided diagnosis framework. However, this task remains a challenging problem due to issues like overlapping and touching objects, shape variation, and background complexity. To address this challenge, we present a novel and effective approach for instance segmentation through the synergistic combination of two deep learning networks (detection and segmentation models) with active shape models. Our method, called softmax-driven active shape model (SD-ASM), uses information from deep neural networks to initialize and evolve a dynamic deformable model. The detection module of the deep network enables the treatment of individual objects separately, while the semantic segmentation map precisely outlines the boundaries of histological structures. Our experiments demonstrate that integrating SD-ASMs into deep learning frameworks significantly enhances instance segmentation performance. We conducted extensive tests using various state-of-the-art architectures on two standard datasets for segmenting crowded objects like cell nuclei -MoNuSeg and CoNIC. To emphasize the versatility of our approach, we applied SD-ASMs in two additional clinical cases: steatosis and tubules segmentation. Once again, the integration of SD-ASM consistently outperformed reference methods, highlighting its effectiveness in accurately segmenting touching objects across multiple clinical scenarios. The proposed approach is inherently generalizable and can be easily applied to other imaging modalities and crowded object segmentation tasks, including fluorescence cellular imaging.