Finetuning a pretrained backbone in the encoder part of an image transformer network has been the traditional approach for the semantic segmentation task. However, such an approach leaves out the semantic context that an image provides during the encoding stage. This paper argues that incorporating semantic information of the image into pretrained hierarchical transformer-based backbones while finetuning improves the performance considerably. To achieve this, we propose SeMask, a simple and effective framework that incorporates semantic information into the encoder with the help of a semantic attention operation. In addition, we use a lightweight semantic decoder during training to provide supervision to the intermediate semantic prior maps at every stage. Our experiments demonstrate that incorporating semantic priors enhances the performance of the established hierarchical encoders with a slight increase in the number of FLOPs. We provide empirical proof by integrating Se-Mask into each variant of the Swin-Transformer as our encoder paired with different decoders. Our framework achieves a new state-of-the-art of 58.22% mIoU on the ADE20K dataset and improvements of over 3% in the mIoU metric on the Cityscapes dataset. The code and checkpoints are publicly available at https://github.com/Picsart-AI-Research/SeMask-Segmentation.

github.io/fcf-inpainting/ LaMa Ours CoModGAN Ours Image with Holes Image with Holes Geometry Structures and Object Boundary Appearance Textures and Repeating PatternsFigure 1: The most challenging issues for advanced image inpainting algorithms fall on generating better structures and textures. Left: LaMa [27] works well for repeating textures but generates fading out boundaries and structures when the holes get larger. Right: CoModGAN [43] with a StyleGAN-based [13] generator achieves impressive geometry structures, but it fails to reuse textures within the image to generate plausible repeating patterns. Our model generates good structures and textures simultaneously better than any state-of-the-arts.