Robotic goalie with 3 ms reaction time at 4% CPU load using event-based dynamic vision sensor

We present a method to predict collisions with objects thrown at a quadrotor using a pair of dynamic vision sensors (DVS). Due to the micro-second temporal resolution of these sensors and the sparsity of their output, the object's trajectory can be estimated with minimal latency. Unlike standard cameras that send frames at a fixed frame rate, a DVS only transmits pixel-level brightness changes ("events") at the time they occur. Our method tracks spherical objects on the image plane using probabilistic trackers that are updated with each incoming event. The object's trajectory is estimated using an Extended Kalman Filter with a mixed state space that allows incorporation of both the object's dynamics and the measurement noise in the image plane. Using error-propagation techniques, we predict a collision if the 3-ellipsoid along the predicted trajectory intersects with a safety sphere around the quadrotor. We experimentally demonstrate that our method allows initiating evasive maneuvers early enough to avoid collisions. Abstract-We present a method to predict collisions with objects thrown at a quadrotor using a pair of dynamic vision sensors (DVS). Due to the micro-second temporal resolution of these sensors and the sparsity of their output, the object's trajectory can be estimated with minimal latency. Unlike standard cameras that send frames at a fixed frame rate, a DVS only transmits pixel-level brightness changes ("events") at the time they occur. Our method tracks spherical objects on the image plane using probabilistic trackers that are updated with each incoming event. The object's trajectory is estimated using an Extended Kalman Filter with a mixed state space that allows incorporation of both the object's dynamics and the measurement noise in the image plane. Using error-propagation techniques, we predict a collision if the 3σ-ellipsoid along the predicted trajectory intersects with a safety sphere around the quadrotor. We experimentally demonstrate that our method allows initiating evasive maneuvers early enough to avoid collisions.

show abstract

“…Table II provide the same results measured on the embedded computer. Our results are in line with the results reported in [9] of (2.2 ± 2.0) ms.…”

Section: B Resultssupporting

confidence: 83%

“…Using two DVS, the authors implemented a pencil-balancing system on a highly-reactive platform free to move on a plane. A robotic goalkeeper with a reaction time of 3 ms was presented in [9].…”

Section: Related Workmentioning

confidence: 99%

Towards evasive maneuvers with quadrotors using dynamic vision sensors

Mueggler

Baumli

Fontana

et al. 2015

2015 European Conference on Mobile Robots (ECMR)

View full text Add to dashboard Cite

show abstract

“…The extreme low-latency of event cameras enabled fast closeloop control (e.g., inverse pendulum balancing (Conradt et al, 2009) and goal keeping with 3 ms reaction time and only 4% CPU utilization (Delbruck and Lang, 2013)). High-frequency visual feedback (>1 kHz) enabled stable manipulator control at micrometer scale (Ni et al, 2012).…”

Section: Event-driven Vision For Robotsmentioning

confidence: 99%

The Event-Driven Software Library for YARP—With Algorithms and iCub Applications

et al. 2018

View full text Add to dashboard Cite

Citation:Glover A, Vasco V, Iacono M and Bartolozzi C (2018) Event-driven (ED) cameras are an emerging technology that sample the visual signal based on changes in the signal magnitude, rather than at a fixed-rate over time. The change in paradigm results in a camera with a lower latency, that uses less power, has reduced bandwidth, and higher dynamic range. Such cameras offer many potential advantages for on-line, autonomous, robots; however, the sensor data do not directly integrate with current "image-based" frameworks and software libraries. The iCub robot uses Yet Another Robot Platform (YARP) as middleware to provide modular processing and connectivity to sensors and actuators. This paper introduces a library that incorporates an event-based framework into the YARP architecture, allowing event cameras to be used with the iCub (and other YARP-based) robots. We describe the philosophy and methods for structuring events to facilitate processing, while maintaining low-latency and real-time operation. We also describe several processing modules made available open-source, and three example demonstrations that can be run on the neuromorphic iCub.

show abstract

“…Early event-based feature trackers were very simple and focused on demonstrating the low-latency and lowprocessing requirements of event-driven vision systems, hence they tracked moving objects as clustered blob-like sources of events [11], [12], [13], [14], [15] or lines [16].…”

Section: B Event-based Tracking Literaturementioning

confidence: 99%

Feature detection and tracking with the dynamic and active-pixel vision sensor (DAVIS)

Tedaldi

Gallego

Mueggler

et al. 2016

2016 Second International Conference on Event-Based Control, Communication, and Signal Processing (EBCCSP)

View full text Add to dashboard Cite

Abstract-Because standard cameras sample the scene at constant time intervals, they do not provide any information in the blind time between subsequent frames. However, for many high-speed robotic and vision applications, it is crucial to provide high-frequency measurement updates also during this blind time. This can be achieved using a novel vision sensor, called DAVIS, which combines a standard camera and an asynchronous event-based sensor in the same pixel array. The DAVIS encodes the visual content between two subsequent frames by an asynchronous stream of events that convey pixellevel brightness changes at microsecond resolution. We present the first algorithm to detect and track visual features using both the frames and the event data provided by the DAVIS. Features are first detected in the grayscale frames and then tracked asynchronously in the blind time between frames using the stream of events. To best take into account the hybrid characteristics of the DAVIS, features are built based on large, spatial contrast variations (i.e., visual edges), which are the source of most of the events generated by the sensor. An eventbased algorithm is further presented to track the features using an iterative, geometric registration approach. The performance of the proposed method is evaluated on real data acquired by the DAVIS.

show abstract

Robotic goalie with 3 ms reaction time at 4% CPU load using event-based dynamic vision sensor

Cited by 183 publications

References 14 publications

Towards evasive maneuvers with quadrotors using dynamic vision sensors

Towards evasive maneuvers with quadrotors using dynamic vision sensors

The Event-Driven Software Library for YARP—With Algorithms and iCub Applications

Feature detection and tracking with the dynamic and active-pixel vision sensor (DAVIS)

Contact Info

Product

Resources

About