Ultimate SLAM? Combining Events, Images, and IMU for Robust Visual SLAM in HDR and High-Speed Scenarios

Abstract

Abstract-Event cameras are bio-inspired vision sensors that output pixel-level brightness changes instead of standard intensity frames. These cameras do not suffer from motion blur and have a very high dynamic range, which enables them to provide reliable visual information during high speed motions or in scenes characterized by high dynamic range. However, event cameras output only little information when the amount of motion is limited, such as in the case of almost still motion. Conversely, standard cameras provide instant and rich information about the environment most of the time (in low-speed and good lighting scenarios), but they fail severely in case of fast motions, or difficult lighting such as high dynamic range or low light scenes. In this paper, we present the first state estimation pipeline that leverages the complementary advantages of these two sensors by fusing in a tightly-coupled manner events, standard frames, and inertial measurements. We show on the publicly available Event Camera Dataset that our hybrid pipeline leads to an accuracy improvement of 130% over event-only pipelines, and 85% over standard-frames-only visual-inertial systems, while still being computationally tractable. Furthermore, we use our pipeline to demonstrate-to the best of our knowledge-the first autonomous quadrotor flight using an event camera for state estimation, unlocking flight scenarios that were not reachable with traditional visual-inertial odometry, such as low-light environments and high-dynamic range scenes.

Long‐duration fully autonomous operation of rotorcraft unmanned aerial systems for remote‐sensing data acquisition

“…As a result, automatic exposure control is important. Improvements can be made by adapting exposure control to the expected brightness level of the observed landing target, or by using a different technology like an event‐based camera which has been shown to be less susceptible to exposure effects (Vidal, Rebecq, Horstschaefer, & Scaramuzza, ).…”

“…Figure 26 shows an example mission data product monitoring the surface temperature of a rock formation at different times of day. (Vidal, Rebecq, Horstschaefer, & Scaramuzza, 2018).…”

Neuromorphic vision: From sensors to event‐based algorithms

“…The next leap in visual IMU fusion came with the introduction of tight coupling of the frame‐based camera, event‐based camera, and the IMU (Vidal et al, ) for pose estimation to leverage the complementary advantages of frame‐based cameras and event‐based cameras. This extended the work of Rebecq et al () to include optical cameras as well for fusion.…”

“…Window size becomes a critical parameter Minimum mean position error of 0.54% and yaw error of 0.03 /m (Vidal et al, 2018) Improved performance by combining standard frames with IMU and event data…”

Neuromorphic Visual Odometry System For Intelligent Vehicle Application With Bio-inspired Vision Sensor

“…The system estimates 6-DOF motion of camera in natural environment by tracking sparse features in event streams. This is the first event based visual odometry Rosinol et al 2018 [16] 3D natural X √ fusion with grayscale camera and IMU system that uses sparse feature points, but it still requires intensity images from conventional camera to detect feature points at first. Compared with the visual odometry system completely based on conventional camera, this system avoids a tremendous computational burden of feature tracking.…”