EV-SegNet: Semantic Segmentation for Event-Based Cameras

Abstract

Event cameras, or Dynamic Vision Sensor (DVS), are very promising sensors which have shown several advantages over frame based cameras. However, most recent work on real applications of these cameras is focused on 3D reconstruction and 6-DOF camera tracking. Deep learning based approaches, which are leading the state-of-the-art in visual recognition tasks, could potentially take advantage of the benefits of DVS, but some adaptations are needed still needed in order to effectively work on these cameras. This work introduces a first baseline for semantic segmentation with this kind of data. We build a semantic segmentation CNN based on state-of-the-art techniques which takes event information as the only input. Besides, we propose a novel representation for DVS data that outperforms previously used event representations for related tasks. Since there is no existing labeled dataset for this task, we propose how to automatically generate approximated semantic segmentation labels for some sequences of the DDD17 dataset, which we publish together with the model, and demonstrate they are valid to train a model for DVS data only. We compare our results on semantic segmentation from DVS data with results using corresponding grayscale images, demonstrating how they are complementary and worth combining.

Video to Events: Recycling Video Datasets for Event Cameras

“…The latter dataset features multiple vehicles in different environments and also provides ego-motion and LIDAR data together with frames and events from a stereo DAVIS setup. A subset of DDD17 was later extended [1] with approximate semantic labels to investigate semantic segmentation with event cameras.…”

“…Later, Maqueda et al [26] proposed an event-frame representation and designed a CNN architecture for steering angle regression on the DDD17 dataset. The same dataset has been modified by Alonso et al [1] to perform semantic segmentation. The availability of MVSEC has spurred research in optical flow [41,42,15] and depth estimation [42,39].…”

“…Thus, they are a viable alternative, or complementary sensor, in conditions that are challenging for standard cameras, such as high-speed and highdynamic-range (HDR) scenarios [19,36,42,33]. 1 However, because the output of event cameras is asynchronous, existing computer vision algorithms developed for standard cameras cannot be directly applied to these data but need to be specifically tailored to leverage event data (for a survey on event cameras and the field of event-based vision, we refer the reader to [12]).…”

“…Recently, novel learning approaches operating on event data have achieved impressive results in scenarios where networks operating on standard cameras fail [26,41,1,42,33,15,34]. Notably, in [34] it was shown that a network trained to reconstruct grayscale intensity frames solely from events can synthesize high framerate videos (> 5, 000 frames per second) of high-speed phenomena (e.g., a bullet shot by gun hitting an object) and can as well render HDR video in challenging lighting conditions (e.g., abrupt transition from dark to bright scene).…”

CDED-Net: Joint Segmentation of Optic Disc and Optic Cup for Glaucoma Screening

“…SegNet is considering fewer trainable parameters because of the elimination of fully connected layers which are responsible for a huge number of parameters. SegNet-Basic architecture [21] is based on VGG-16 architecture (13 convolutional layers) without any skip connections which guarantees the feature-reuse policy to reduce the vanishing gradient problem [47]. As OD and OC are with few pixels in the image, so during the continuous convolutional process the features of OC can vanish completely.…”