Event-Based Robotic Grasping Detection With Neuromorphic Vision Sensor and Event-Grasping Dataset

Abstract

Robotic grasping plays an important role in the field of robotics. The current state-of-the-art robotic grasping detection systems are usually built on the conventional vision, such as the RGB-D camera. Compared to traditional frame-based computer vision, neuromorphic vision is a small and young community of research. Currently, there are limited event-based datasets due to the troublesome annotation of the asynchronous event stream. Annotating large scale vision datasets often takes lots of computation resources, especially when it comes to troublesome data for video-level annotation. In this work, we consider the problem of detecting robotic grasps in a moving camera view of a scene containing objects. To obtain more agile robotic perception, a neuromorphic vision sensor ( Dynamic and Active-pixel Vision Sensor, DAVIS ) attaching to the robot gripper is introduced to explore the potential usage in grasping detection. We construct a robotic grasping dataset named Event-Grasping dataset with 91 objects. A spatial-temporal mixed particle filter (SMP Filter) is proposed to track the LED-based grasp rectangles, which enables video-level annotation of a single grasp rectangle per object. As LEDs blink at high frequency, the Event-Grasping dataset is annotated at a high frequency of 1 kHz. Based on the Event-Grasping dataset, we develop a deep neural network for grasping detection that considers the angle learning problem as classification instead of regression. The method performs high detection accuracy on our Event-Grasping dataset with 93% precision at an object-wise level split. This work provides a large-scale and well-annotated dataset and promotes the neuromorphic vision applications in agile robot.

Unsupervised Adaptive Multi-Object Tracking-by-Clustering Algorithm With a Bio-Inspired System

“…In [58] the object tracking problem is accomplished by training a binary classifier with statistical bootstrapping. Recently, in [59] a spatial-temporal mixed particle filter (SMP Filter) is proposed to track LED-based rectangles. In [49], a Restricted Spatiotemporal Particle Filter (RSPF) tracking algorithm is presented, and evaluated tracking fingers.…”

Keypoint-Based Robotic Grasp Detection Scheme in Multi-Object Scenes

“…Depierre et al [ 21 ] proposed a neural network with a scorer which evaluates the graspability of a given position and introduce a novel loss function which correlates regression of grasp parameters with graspability score. Based on the Event-Stream dataset, Li et al [ 22 ] develop a deep neural network for grasping detection which consider the angle learning problem as classification instead of regression. This work provides a large-scale and well-annotated dataset, and promotes the neuromorphic vision applications in robot grasp.…”

Evolutionary vs imitation learning for neuromorphic control at the edge*

“…Neuromorphic computers have several properties that make them attractive for edge computing applications, primarily low power operation, robustness and resilience, and adaptability and plasticity. There have been several compelling demonstrations of neuromorphic computers for edge applications, including for keyword spotting [7], robotics [3,25,35,49], medical applications [5], and intelligent engine control [45]. However, there are many different options for neuromorphic training algorithms, and it is not always clear what the best algorithm is for a given application.…”