Polarization-driven semantic segmentation via efficient attention-bridged fusion

Abstract

Semantic Segmentation (SS) is promising for outdoor scene perception in safetycritical applications like autonomous vehicles, assisted navigation and so on. However, traditional SS is primarily based on RGB images, which limits the reliability of SS in complex outdoor scenes, where RGB images lack necessary information dimensions to fully perceive unconstrained environments. As preliminary investigation, we examine SS in an unexpected obstacle detection scenario, which demonstrates the necessity of multimodal fusion. Thereby, in this work, we present EAFNet, an Efficient Attention-bridged Fusion Network to exploit complementary information coming from different optical sensors. Specifically, we incorporate polarization sensing to obtain supplementary information, considering its optical characteristics for robust representation of diverse materials. By using a single-shot polarization sensor, we build the first RGB-P dataset which consists of 394 annotated pixel-aligned RGB-Polarization images. A comprehensive variety of experiments shows the effectiveness of EAFNet to fuse polarization and RGB information, as well as the flexibility to be adapted to other sensor combination scenarios.

Trans4Trans: Efficient Transformer for Transparent Object and Semantic Scene Segmentation in Real-World Navigation Assistance

“…Moreover, polarization cues [43] and reflection priors [44] were often explored for transparency perception. For instance, Xiang et al [45] built a polarization-driven semantic segmentation architecture by bridging RGB and polarization dimensions dynamically using efficient attention connections, which considers the optical features of polarimetric information for robust representation of diverse materials and lifts the performance of classes with polarization properties like glass.…”

“…Recently, large-scale transparent object segmentation datasets emerge [14], [24], [42], [45], [46]. Mei et al [14] constructed the glass detection dataset in daily-life scenes.…”

Capturing Omni-Range Context for Omnidirectional Segmentation

“…A second branch of work takes advantage of recent channel/spatial attention mechanisms [20,21,62,65] to exploit global context [29,86], dimension-wise priors [8,72] and cross-modal features [55,67,88]. More recently, inspired by non-local blocks in recognition tasks [64], DANet [17] and OCNet [83] adopt self-attention [60] to capture either associations between any pair of pixels/channels or dense object context, respectively.…”

Trans4Trans: Efficient Transformer for Transparent Object Segmentation to Help Visually Impaired People Navigate in the Real World

“…Polarization cues [31] and reflection priors [36] are also frequently explored for transparency perception. For example, Xiang et al [68] propose a polarization-driven semantic segmentation architecture by adaptively bridging RGB and polarization dimensions, which significantly lift the performance of classes with polarization properties like glass.…”