Polyp-PVT: Polyp Segmentation with Pyramid Vision Transformers

Abstract

Most polyp segmentation methods use convolutional neural networks (CNNs) as their backbone, leading to two key issues when exchanging information between the encoder and decoder: (1) taking into account the differences in contribution between differentlevel features, and (2) designing an effective mechanism for fusing these features. Unlike existing CNN-based methods, we adopt a transformer encoder, which learns more powerful and robust representations. In addition, considering the image acquisition influence and elusive properties of polyps, we introduce three standard modules, including a cascaded fusion module (CFM), a camouflage identification module (CIM), and a similarity aggregation module (SAM). Among these, the CFM is used to collect the semantic and location information of polyps from high-level features; the CIM is applied to capture polyp information disguised in low-level features, and the SAM extends the pixel features of the polyp area with high-level semantic position information to the entire polyp area, thereby effectively fusing cross-level features. The proposed model, named Polyp-PVT, effectively suppresses noises in the features and significantly improves their expressive capabilities. Extensive experiments on five widely adopted datasets show that the proposed model is more robust to various challenging situations (e.g., appearance changes, small objects, and rotation) than existing representative methods. The proposed model is available at https://github.com/DengPingFan/Polyp-PVT.

Improved dual-aggregation polyp segmentation network combining a pyramid vision transformer with a fully convolutional network

“…Due to only a single scale of output feature map with low resolution, it was challenging to directly adapt it to polyp segmentation task. Based this consideration, several methods [ 33 – 37 ] incorporated the pyramid structure in CNN into the design of Transformers, presenting a hierarchical Transformer with different stages. Dong et al [ 33 ] utilized a pyramid vision Transformer (PVT) as backbone encoder and presented a polyp segmentation architecture called Polyp-PVT.…”

“…Based this consideration, several methods [ 33 – 37 ] incorporated the pyramid structure in CNN into the design of Transformers, presenting a hierarchical Transformer with different stages. Dong et al [ 33 ] utilized a pyramid vision Transformer (PVT) as backbone encoder and presented a polyp segmentation architecture called Polyp-PVT. Tang et al [ 34 ] proposed a Dual-Aggregation Transformer Network (DuAT) to segment polyp regions, which adapted the PVT as the encoder for capturing richer feature cues.…”

“…Although some methods focused on capturing information from multiple scales and multiple receptive fields on the basis of the atrous spatial pyramid pooling (ASPP) block or DenseASPP, they would produce many extra parameters and computations, not to mention over-fitting. Different from DCNN-based methods, Transformer-based methods such as Polyp-PVT [ 33 ], SSFormer [ 35 ], and ESFPNet [ 37 ] etc., were able to perceive semantic distinctions from a global aspect via adaptively modeling long-distance relationships, and possessed stronger feature representation abilities by way of capturing global contextual information. Notwithstanding demonstrating impressive performance over DCNN-based alternatives, grabbing long-distance dependencies could destroy portion of task-specific critical local features and fail to model interactive features from the neighborhood, which could cause overly smooth segmentation for small/tiny polyps and vague boundaries between polyps.…”

Dual‐branch feature extraction network combined with Transformer and CNN for polyp segmentation

“…Although the convolution operation of CNN can capture image details well, it has some limitations in accessing global and long‐range semantic information 39 . Many researchers 20,40–43 combined Transformer and CNN to utilize complementary advantages between each other to obtain effective segmentation performance. TransUNet 20 was the first work that applied Transformer layers to the U‐Net structure for medical image segmentation.…”

“…The segmentation performance, parameter number, and inference speed have the improvements to some extent. Poly‐PVT 41 designed a pyramid vision transformer (PVT) encoder to extract multi‐scale long‐range dependencies features from input images. Three simple modules based on CNN, that is, cascaded fusion module, camouflage identification module, and similarity aggregation module were introduced to achieve superior segmentation results.…”