Event-Based Asynchronous Sparse Convolutional Networks

Abstract

Event cameras are bio-inspired sensors that respond to perpixel brightness changes in the form of asynchronous and sparse "events". Recently, pattern recognition algorithms, such as learning-based methods, have made significant progress with event cameras by converting events into synchronous dense, image-like representations and applying traditional machine learning methods developed for standard cameras. However, these approaches discard the spatial and temporal sparsity inherent in event data at the cost of higher computational complexity and latency. In this work, we present a general framework for converting models trained on synchronous image-like event representations into asynchronous models with identical output, thus directly leveraging the intrinsic asynchronous and sparse nature of the event data. We show both theoretically and experimentally that this drastically reduces the computational complexity and latency of high-capacity, synchronous neural networks without sacrificing accuracy. In addition, our framework has several desirable characteristics: (i) it exploits spatio-temporal sparsity of events explicitly, (ii) it is agnostic to the event representation, network architecture, and task, and (iii) it does not require any train-time change, since it is compatible with the standard neural networks' training process. We thoroughly validate the proposed framework on two computer vision tasks: object detection and object recognition. In these tasks, we reduce the computational complexity up to 20 times with respect to high-latency neural networks. At the same time, we outperform state-ofthe-art asynchronous approaches up to 24% in prediction accuracy.

Data‐Driven Technology in Event‐Based Vision

“…Event cameras output a stream of events represented as a tuple of location, timestamp, and polarity of the intensity change. Owing to the unique working principle, event data is endowed with the spatial and temporal sparsity nature [4], corresponding to its high efficiency. On the other side, information extraction from event data plays a crucial role for further analysis in event-based vision.…”

“…Since an event alone contains little information, each incoming event is usually coupled with extra information from past events for further estimation. Requiring expert knowledge, feature descriptors and measurement update functions [4] require to be handcrafted and task-oriented in these methods, which slows down their widespread adoption in high-level vision such as recognition and segmentation. Despite minimal latency, these methods perform redundant computation owing to frequent system state update, with accuracy sensitive to algorithm parameters.…”

“…Methods above provide solutions with both advantages and disadvantages, and recent research makes optimization by combining complementary advantages that each category has to offer. Current innovation architectures tailored to high-rate, variablelength, and nonuniform event stream have been proposed to balance the trade-off between accuracy and efficiency while protecting the spatio-temporal sparsity [4,43,44]. Generally speaking, with high capacity neural networks, features are automatically learned from data by optimizing the corresponding object function in data-driven approaches, exempt from handcrafted feature descriptors.…”

“…Methods above contribute solutions with both advantages and disadvantages and recent research aims at integrating the possibilities each category has to offer, as illustrated in Table 1. Current innovation architectures tailored to high-rate variablelength event streams have emerged as a promising candidate to balance the trade-off between accuracy and efficiency while protecting the spatio-temporal sparsity [4,44]. Authors of [44] presented a deep hybrid neural network named Spike-FlowNet for optical flow estimation.…”

“…e combination of SNNs and ANNs improves its computation efficiency while maintaining performance. Researchers in [4] brought the concept of event-based asynchronous sparse convolutional networks into public sight. To be concrete, a framework that transforms models trained with image-like event representations into asynchronous models is raised to better leverage the asynchronous and sparse nature of events, surmounting the original limitation of deep neural networks in event-based vision.…”