Feature detection and tracking with the dynamic and active-pixel vision sensor (DAVIS)

Tedaldi, David; Gallego, Guillermo; Mueggler, Elias; Scaramuzza, Davide

doi:10.1109/ebccsp.2016.7605086

Cited by 95 publications

(74 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Afterwards, they tracked these feature corners based on both frames using Kanade-Lucas-Tomasi (KLT) tracker [13]. In [14]- [16], they first extracted intensity-corners based on intensity-frames using the Harris detector [17], and then tracked these intensity-corners based on event-streams.…”

Section: A Non-synchronous Event-based Corner Detectionmentioning

confidence: 99%

FA-Harris: A Fast and Asynchronous Corner Detector for Event Cameras

Shi²,

Zhang³

et al. 2019

2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

View full text Add to dashboard Cite

Recently, the emerging bio-inspired event cameras have demonstrated potentials for a wide range of robotic applications in dynamic environments. In this paper, we propose a novel fast and asynchronous event-based corner detection method which is called FA-Harris. FA-Harris consists of several components, including an event filter, a Global Surface of Active Events (G-SAE) maintaining unit, a corner candidate selecting unit, and a corner candidate refining unit. The proposed G-SAE maintenance algorithm and corner candidate selection algorithm greatly enhance the real-time performance for corner detection, while the corner candidate refinement algorithm maintains the accuracy of performance by using an improved event-based Harris detector. Additionally, FA-Harris does not require artificially synthesized event-frames and can operate on asynchronous events directly. We implement the proposed method in C++ and evaluate it on public Event Camera Datasets. The results show that our method achieves approximately 8× speed-up when compared with previously reported event-based Harris detector, and with no compromise on the accuracy of performance.

show abstract

Section: A Non-synchronous Event-based Corner Detectionmentioning

confidence: 99%

FA-Harris: A Fast and Asynchronous Corner Detector for Event Cameras

Shi²,

Zhang³

et al. 2019

2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

View full text Add to dashboard Cite

show abstract

“…Recently, we presented a hybrid method for feature detection and tracking for the DAVIS [18]. The method first detects and extracts features in the frames and then tracks them using only the events.…”

Section: A Event-based Feature Detection and Trackingmentioning

confidence: 99%

“…The method first detects and extracts features in the frames and then tracks them using only the events. In the present paper, we improve [18] by (i) taking into account the observation that nearby pixels typically observe events at roughly the same time and (ii) introducing a tracking refinement step that works on a slower timescale to avoid drift. Furthermore, we improve the tracking speed and add dynamic reinitialization of new features (e.g., in new areas of the scene or when features are lost).…”

Section: A Event-based Feature Detection and Trackingmentioning

confidence: 99%

See 1 more Smart Citation

Low-latency visual odometry using event-based feature tracks

Kueng

Mueggler

Gallego

et al. 2016

2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

Self Cite

162

159

View full text Add to dashboard Cite

New vision sensors, such as the Dynamic and Active-pixel Vision sensor (DAVIS), incorporate a conventional camera and an event-based sensor in the same pixel array. These sensors have great potential for robotics because they allow us to combine the benefits of conventional cameras with those of eventbased sensors: low latency, high temporal resolution, and high dynamic range. However, new algorithms are required to exploit the sensor characteristics and cope with its unconventional output, which consists of a stream of asynchronous brightness changes (called "events") and synchronous grayscale frames. In this paper, we present a lowlatency visual odometry algorithm for the DAVIS sensor using event-based feature tracks. Features are first detected in the grayscale frames and then tracked asynchronously using the stream of events. The features are then fed to an event-based visual odometry algorithm that tightly interleaves robust pose optimization and probabilistic mapping. We show that our method successfully tracks the 6-DOF motion of the sensor in natural scenes. This is the first work on event-based visual odometry with the DAVIS sensor using feature tracks. Low-Latency Visual Odometry using Event-based Feature Tracks Beat Kueng, Elias Mueggler, Guillermo Gallego and Davide ScaramuzzaAbstract-New vision sensors, such as the Dynamic and Active-pixel Vision sensor (DAVIS), incorporate a conventional camera and an event-based sensor in the same pixel array. These sensors have great potential for robotics because they allow us to combine the benefits of conventional cameras with those of event-based sensors: low latency, high temporal resolution, and high dynamic range. However, new algorithms are required to exploit the sensor characteristics and cope with its unconventional output, which consists of a stream of asynchronous brightness changes (called "events") and synchronous grayscale frames. In this paper, we present a lowlatency visual odometry algorithm for the DAVIS sensor using event-based feature tracks. Features are first detected in the grayscale frames and then tracked asynchronously using the stream of events. The features are then fed to an event-based visual odometry algorithm that tightly interleaves robust pose optimization and probabilistic mapping. We show that our method successfully tracks the 6-DOF motion of the sensor in natural scenes. This is the first work on event-based visual odometry with the DAVIS sensor using feature tracks. MULTIMEDIA MATERIALA video attachment to this work is available on the authors' webpage.

show abstract

“…Event cameras, such as the Dynamic Vision Sensor (DVS) [1] posses outstanding properties compared to traditional cameras: very high dynamic range (140 dB vs. 60 dB), high temporal resolution (in the order of µs), and do not suffer from motion blur. Hence, event cameras have a large potential to tackle challenging scenarios for standard cameras (such as high speed and high dynamic range) in tracking [2][3][4][5][6][7][8][9], depth estimation [10][11][12][13][14][15][16][17][18][19], Simultaneous Localization and Mapping [20][21][22][23][24][25][26][27], and recognition [28][29][30][31][32], among other applications. However, novel methods are required to process the unconventional output of these sensors in order to unlock their potential.…”

Section: Introductionmentioning

confidence: 99%

Focus Is All You Need: Loss Functions for Event-Based Vision

Gallego

Gehrig

Scaramuzza

2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Self Cite

120

116

View full text Add to dashboard Cite

θ I(x; θ) Parameters Warped Events Focus score Warping along point trajectories X [pix] Y [pix] time [s] X [pix] Y [pix] Measure event alignment Input Events θFigure 1: Motion Compensation Framework. Events in a space-time window are warped according to point trajectories described by motion parameters θ, resulting in an image of warped events (IWE) I(x; θ). Then, a focus loss function of I measures how well events are aligned along the point trajectories. This work proposes multiple focus loss functions for event alignment (last block in the figure and Table 1) for tasks such as rotational motion, depth and optical flow estimation. AbstractEvent cameras are novel vision sensors that output pixellevel brightness changes ("events") instead of traditional video frames. These asynchronous sensors offer several advantages over traditional cameras, such as, high temporal resolution, very high dynamic range, and no motion blur. To unlock the potential of such sensors, motion compensation methods have been recently proposed. We present a collection and taxonomy of twenty two objective functions to analyze event alignment in motion compensation approaches (Fig. 1). We call them focus loss functions since they have strong connections with functions used in traditional shapefrom-focus applications. The proposed loss functions allow bringing mature computer vision tools to the realm of event cameras. We compare the accuracy and runtime performance of all loss functions on a publicly available dataset, and conclude that the variance, the gradient and the Laplacian magnitudes are among the best loss functions. The applicability of the loss functions is shown on multiple tasks: rotational motion, depth and optical flow estimation. The proposed focus loss functions allow to unlock the outstanding properties of event cameras. †

show abstract

Feature detection and tracking with the dynamic and active-pixel vision sensor (DAVIS)

Cited by 95 publications

References 21 publications

FA-Harris: A Fast and Asynchronous Corner Detector for Event Cameras

FA-Harris: A Fast and Asynchronous Corner Detector for Event Cameras

Low-latency visual odometry using event-based feature tracks

Focus Is All You Need: Loss Functions for Event-Based Vision

Contact Info

Product

Resources

About