Neuromorphic Event-Based Slip Detection and Suppression in Robotic Grasping and Manipulation

Muthusamy, Rajkumar; Huang, Xiaoqian; Zweiri, Yahya; Seneviratne, Lakmal; Gan, Dongming

doi:10.1109/access.2020.3017738

Cited by 36 publications

(20 citation statements)

References 65 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Therefore, the unique property of DAVIS becomes indispensable to improve the performance for grasping in sorting applications. To that end, few works employed DAVIS for tackling grasping behaviors, such as dynamic force estimation [14] and incipient slippages detection and suppression [15,16]. In this work, we explore and study how the event based tactile sensor with occluded skin can be effective in contact-level classification, especially in robotic sorting applications.…”

Section: Introductionmentioning

confidence: 99%

Neuromorphic Vision Based Contact-Level Classification in Robotic Grasping Applications

Huang

Muthusamy

Hassan

et al. 2020

Sensors

Self Cite

View full text Add to dashboard Cite

In recent years, robotic sorting is widely used in the industry, which is driven by necessity and opportunity. In this paper, a novel neuromorphic vision-based tactile sensing approach for robotic sorting application is proposed. This approach has low latency and low power consumption when compared to conventional vision-based tactile sensing techniques. Two Machine Learning (ML) methods, namely, Support Vector Machine (SVM) and Dynamic Time Warping-K Nearest Neighbor (DTW-KNN), are developed to classify material hardness, object size, and grasping force. An Event-Based Object Grasping (EBOG) experimental setup is developed to acquire datasets, where 243 experiments are produced to train the proposed classifiers. Based on predictions of the classifiers, objects can be automatically sorted. If the prediction accuracy is below a certain threshold, the gripper re-adjusts and re-grasps until reaching a proper grasp. The proposed ML method achieves good prediction accuracy, which shows the effectiveness and the applicability of the proposed approach. The experimental results show that the developed SVM model outperforms the DTW-KNN model in term of accuracy and efficiency for real time contact-level classification.

show abstract

Section: Introductionmentioning

confidence: 99%

Neuromorphic Vision Based Contact-Level Classification in Robotic Grasping Applications

Huang

Muthusamy

Hassan

et al. 2020

Sensors

Self Cite

View full text Add to dashboard Cite

show abstract

“…Event-based neuromorphic vision sensors have become significantly popular recently and introduce a paradigm in computer vision applications for (VBM) systems [11,[51][52][53][54][55][56]. Thanks to the low latency and low power consumption of the event-based sensor, an event-based frame approach is proposed to measure the contact force in grasping applications by attaching the event-based sensor to an elastic material in [11,51] and [53,56] for incipient slip detection. [11,54,55] use the event-based sensor for force estimation.…”

Section: Algorithms For Eventmentioning

confidence: 99%

Mixed Event-Frame Vision System for Daytime Preceding Vehicle Taillight Signal Measurement Using Event-Based Neuromorphic Vision Sensor

Liu

Chen

et al. 2022

Journal of Advanced Transportation

View full text Add to dashboard Cite

An important aspect of the perception system for intelligent vehicles is the detection and signal measurement of vehicle taillights. In this work, we present a novel vision-based measurement (VBM) system, using an event-based neuromorphic vision sensor, which is able to detect and measure the vehicle taillight signal robustly. To the best of our knowledge, it is for the first time the neuromorphic vision sensor is paid attention to for utilizing in the field of vehicle taillight signal measurement. The event-based neuromorphic vision sensor is a bioinspired sensor that records pixel-level intensity changes, called events, as well as the whole picture of the scene. The events naturally respond to illumination changes (such as the ON and OFF state of taillights) in the scene with very low latency. Moreover, the property of a higher dynamic range increases the sensor sensitivity and performance in poor lighting conditions. In this paper, we consider an event-driven solution to measure vehicle taillight signals. In contrast to most existing work that relies purely on standard frame-based cameras for the taillight signal measurement, the presented mixed event/frame system extracts the frequency domain features from the spatial and temporal signal of each taillight region and measures the taillight signal by combining the active-pixel sensor (APS) frames and dynamic vision sensor (DVS) events. A thresholding algorithm and a learned classifier are proposed to jointly achieve the brake-light and turn-light signal measurement. Experiments with real traffic scenes demonstrate the performance of measuring taillight signals under different traffic conditions with a single event-based neuromorphic vision sensor. The results show the high potential of the event-based neuromorphic vision sensor being used for optical signal measurement applications, especially in dynamic environments.

show abstract

“…The sensing mechanism of event-based cameras allows detecting transient microsecond level changes in dynamic scene without a global shutter, which is fundamentally different from frame-based cameras. In our recent works [34]- [36], we exploited such potential of event cameras (1) to passively detect incipient and gross slips of a grasped object at a 2KHz sampling rate and suppressed such slips with intelligent grasp controller [34]. (2) to measure contact level forces irrespective of the object size using deep learning methods [35]; (3) to classify grasped objects in the contact level along with machine learning methods for sorting applications [36].…”

Section: B Contributionsmentioning

confidence: 99%

Neuromorphic Eye-in-Hand Visual Servoing

et al. 2021

Self Cite

View full text Add to dashboard Cite

Robotic vision plays a major role in factory automation to service robot applications. However, the traditional use of frame-based cameras sets a limitation on continuous visual feedback due to their low sampling rate, poor performance in low light conditions and redundant data in real-time image processing, especially in the case of high-speed tasks. Neuromorphic event-based vision is a recent technology that gives human-like vision capabilities such as observing the dynamic changes asynchronously at a high temporal resolution (1µs) with low latency and wide dynamic range. In this paper, for the first time, we present a purely event-based visual servoing method using a neuromorphic camera in an eye-in-hand configuration for the grasping pipeline of a robotic manipulator. We devise three surface layers of active events to directly process the incoming stream of events from relative motion. A purely event-based approach is used to detect corner features, localize them robustly using heatmaps and generate virtual features for tracking and grasp alignment. Based on the visual feedback, the motion of the robot is controlled to make the temporal upcoming event features converge to the desired event in Spatiotemporal space. The controller switches its operation such that it explores the workspace, reaches the target object and achieves a stable grasp. The event-based visual servoing (EBVS) method is comprehensively studied and validated experimentally using a commercial robot manipulator in an eye-in-hand configuration for both static and dynamic targets. Experimental results show superior performance of the EBVS method over frame-based vision, especially in high-speed operations and poor lighting conditions. As such, EBVS overcomes the issues of motion blur, lighting and exposure timing that exist in conventional frame-based visual servoing methods.

show abstract

Neuromorphic Event-Based Slip Detection and Suppression in Robotic Grasping and Manipulation

Cited by 36 publications

References 65 publications

Neuromorphic Vision Based Contact-Level Classification in Robotic Grasping Applications

Neuromorphic Vision Based Contact-Level Classification in Robotic Grasping Applications

Mixed Event-Frame Vision System for Daytime Preceding Vehicle Taillight Signal Measurement Using Event-Based Neuromorphic Vision Sensor

Neuromorphic Eye-in-Hand Visual Servoing

Contact Info

Product

Resources

About