Tripartite real-time semantic segmentation network with scene commonality

Abstract

The two-branch real-time semantic segmentation network can quickly acquire lowlevel details and high-level semantics. However, the large contextual gap between them results in adverse impact on their fusion, and limits the further improvement of real-time segmentation accuracy. This paper proposes a tripartite real-time semantic segmentation network with scene commonality (TriSCNet) to address this problem. First, we add a parallel scene commonality branch based on the current two-branch architecture to learn intrinsic common features in similar street scene images, such as the spatial location distribution of various objects and the internal connections between them at the semantic level. Further, with the guidance of commonality, we propose an external branch attention module to enrich and enhance the feature information of traditional two branches. Finally, we utilize an alignment and selective fusion module to correct the misaligned context in the semantic branch and highlight the essential spatial information in the detailed branch. Our proposed TriSCNet achieves an excellent trade-off between accuracy and speed, yielding 77.9% mIOU at 67.2 FPS on Cityscapes test set and 75.8% mIOU at 127.4 FPS on CamVid test set, respectively.