ESVIO: Event-Based Stereo Visual-Inertial Odometry

Abstract

The emerging event cameras are bio-inspired sensors that can output pixel-level brightness changes at extremely high rates, and event-based visual-inertial odometry (VIO) is widely studied and used in autonomous robots. In this paper, we propose an event-based stereo VIO system, namely ESVIO. Firstly, we present a novel direct event-based VIO method, which fuses events’ depth, Time-Surface images, and pre-integrated inertial measurement to estimate the camera motion and inertial measurement unit (IMU) biases in a sliding window non-linear optimization framework, effectively improving the state estimation accuracy and robustness. Secondly, we design an event-inertia semi-joint initialization method, through two steps of event-only initialization and event-inertia initial optimization, to rapidly and accurately solve the initialization parameters of the VIO system, thereby further improving the state estimation accuracy. Based on these two methods, we implement the ESVIO system and evaluate the effectiveness and robustness of ESVIO on various public datasets. The experimental results show that ESVIO achieves good performance in both accuracy and robustness when compared with other state-of-the-art event-based VIO and stereo visual odometry (VO) systems, and, at the same time, with no compromise to real-time performance.

Application of Event Cameras and Neuromorphic Computing to VSLAM: A Survey

“…Conventional cameras record a sequence of images at a predetermined frames per second (fps) rate, capturing intensity values for every pixel in every frame. On the other hand, event cameras record continuous-time event data, timestamped with microsecond resolution, with an event representing a detected change in pixel brightness [9,17,19]. Each pixel continuously updates the log intensity, and this is monitored for any notable changes in its value.…”

“…Following those limitations, research in emerging technologies of event cameras has evolved to attempt to address the issues of traditional VSLAM. The advent of novel concepts and the production of bio-inspired visual sensors and processors through developments in neuroscience and neuromorphic technologies have brought a radical change in the processes of artificial visual systems [9,16,17]. An event camera (also known as a Dynamic Vision Sensor (DVS) or neuromorphic camera) operates very differently from conventional frame-based cameras; it only generates an output (in the form of timestamped events or spikes) when there are changes in the brightness of a scene [9,12,16,18,19].…”

“…An event camera (also known as a Dynamic Vision Sensor (DVS) or neuromorphic camera) operates very differently from conventional frame-based cameras; it only generates an output (in the form of timestamped events or spikes) when there are changes in the brightness of a scene [9,12,16,18,19]. Compared to regular cameras, event cameras have a greater dynamic range, reduced latency, higher temporal resolution, and significantly lower power consumption and bandwidth usage [3,9,12,13,17,18,[20][21][22][23]. However, sensors based on these principles are relatively new to the market and their integration poses some challenges as new algorithms are needed as existing approaches are not directly applicable.…”

Artificial intelligence-based spatio-temporal vision sensors: applications and prospects

“…Xie et al (Xie et al, 2018) used DAVIS240 to build a binocular stereo vision system, and extracted local features from the pulse stream for stereo matching to calculate depth information. In addition, Liu et al (Liu et al, 2023a) constructed a multi-view 3D reconstruction of a monocular camera for the first time for pulse flow, which used an event-based scan plane method to generate a time difference map to estimate depth information. Wang et al (Wang et al, 2022e) extracted the temporal surface features of the pulse signal and established an optimization function to calculate the disparity, which was used for multi-view 3D reconstruction of a monocular camera.…”