Bimodal SegNet: Instance Segmentation Fusing Events and RGB Frames for Robotic Grasping

Abstract

Object segmentation for robotic grasping under dynamic conditions often faces challenges such as occlusion, low light conditions, motion blur and object size variance. To address these challenges, we propose a Deep Learning network that fuses two types of visual signals, event-based data and RGB frame data. The proposed Bimodal SegNet network has two distinct encoders, one for each signal input and a spatial pyramidal pooling with atrous convolutions. Encoders capture rich contextual information by pooling the concatenated features at different resolutions while the decoder obtains sharp object boundaries. The evaluation of the proposed method undertakes five unique image degradation challenges including occlusion, blur, brightness, trajectory and scale variance on the Event-based Segmentation (ESD) Dataset. The evaluation results show a 6-10% segmentation accuracy improvement over state-of-the-art methods in terms of mean intersection over the union and pixel accuracy. The model code is available at https://github.com/sanket0707/Bimodal-SegNet.git

Asynchronous Events-based Panoptic Segmentation using Graph Mixer Neural Network

“…Eventbased methods have advantages as they can handle motion blur and high dynamic range [2] but may require labeled images such as the Event-based Semantic Segmentation (ESS) method [28] and have limitations in segmenting small objects [3]. Multiple modalities can be integrated to leverage complementarity, with CMX using transformer-based architecture [14] and Bimodal SegNet [13] fusing RGB with event frames. Although these methods demonstrate promising results, their limitations include overlooking the high temporal resolution of event-based data.…”

Event Cameras in Automotive Sensing: A Review

“…Semantic Segmentation [169], [170], [171] Advancements in semantic segmentation using CNNs, knowledge distillation, and unsupervised domain adaptation. Instance Segmentation [172], [173], [174] Techniques for instance segmentation fusing event-based data with RGB frames, using posterior attention modules, and modality translation and fusion. Efficient Semantic Segmentation [175] Introduction of a computationally efficient semantic segmentation approach using spiking encoder-decoder network.…”