Attention-Based Transformers for Instance Segmentation of Cells in Microstructures

Abstract

Detecting and segmenting object instances is a common task in biomedical applications. Examples range from detecting lesions on functional magnetic resonance images, to the detection of tumours in histopathological images and extracting quantitative single-cell information from microscopy imagery, where cell segmentation is a major bottleneck. Attention-based transformers are state-of-the-art in a range of deep learning fields. They have recently been proposed for segmentation tasks where they are beginning to outperform other methods. We present a novel attention-based cell detection transformer (Cell-DETR) for direct end-to-end instance segmentation. While the segmentation performance is on par with a state-of-the-art instance segmentation method, Cell-DETR is simpler and faster. We showcase the method's contribution in a the typical use case of segmenting yeast in microstructured environments, commonly employed in systems or synthetic biology. For the specific use case, the proposed method surpasses the state-of-the-art tools for semantic segmentation and additionally predicts the individual object instances. The fast and accurate instance segmentation performance increases the experimental information yield for a posteriori data processing and makes online monitoring of experiments and closed-loop optimal experimental design feasible. Code and data samples are available at https://git.rwth-aachen. de/bcs/projects/cell-detr.git.

Medical Image Segmentation using Squeeze-and-Expansion Transformers

“…Although DETR is also explored to do panoptic segmentation [Kirillov et al, 2019], it adopts a two-stage approach which is not applicable to medical image segmentation. A followup work of DETR, Cell-DETR [Prangemeier et al, 2020] also employs transformer for biomedical image segmentation, but its architecture is just a simplified DETR, lacking novel components like our Squeeze-and-Expansion transformer. Most recently, SETR [Zheng et al, 2021] and TransU-Net [Chen et al, 2021] were released concurrently or after our paper submission.…”

“…Automated Medical image segmentation, i.e., automated delineation of anatomical structures and other regions of interest (ROIs), is an important step in computer-aided diagnosis; for example it is used to quantify tissue volumes, extract key quantitative measurements, and localize pathology [Schlemper et al, 2019;Orlando et al, 2020]. Good segmentation demands the model to see the big picture and fine details at the same time, i.e., learn image features that incorporate large context while keep high spatial resolutions to output finegrained segmentation masks.…”

“…Since the advent of U-Net [Ronneberger et al, 2015], it has shown excellent performance across medical image segmentation tasks. A U-Net consists of an encoder and a decoder, in which the encoder progressively downsamples the features and generates coarse contextual features that focus on contextual patterns, and the decoder progressively upsamples the contextual features and fuses them with fine-grained local visual features.…”

“…Many improvements of U-Net have been proposed. A few typical variants include: U-Net++ [Zhou et al, 2018] and U-Net 3+ [Huang et al, 2020], in which more complicated skip connections are added to better utilize multi-scale contextual features; attention U-Net [Schlemper et al, 2019], which employs attention gates to focus on foreground regions; 3D U-Net [Çiçek et al, 2016] and V-Net [Milletari et al, 2016], which extend U-Net to 3D images, such as MRI volumes; Eff-UNet [Baheti et al, 2020], which replaces the encoder of U-Net with a pretrained EfficientNet [Tan and Le, 2019].…”

DeLTA 2.0: A deep learning pipeline for quantifying single-cell spatial and temporal dynamics

“…Software based on traditional image analysis techniques such as Schnitzcells (17), Oufti (18), or SuperSegger (19) all require significant user input and post-processing. A few recent studies have proposed deep learning models for bacterial or yeast cell segmentation (6,8,10,20), however to our knowledge there is no integrated segmentation and tracking pipeline for two-dimensional timelapse analysis of bacteria.…”