Event-Driven Visual-Tactile Sensing and Learning for Robots

Taunyazov, Tasbolat; Sng, Weicong; Lim, Brian Y.; See, Hian-Hian; Kuan, Jethro; Ansari, Abdul Fatir; Tee, Benjamin C. K.; Soh, Harold

doi:10.15607/rss.2020.xvi.020

Cited by 82 publications

(81 citation statements)

References 50 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…SLAYER with Loihi has also been used for tactile digit recognition by See et al [38] and for sensor fusion because of the ease of combining modalities in the spike domain. Ceolini et al [39] combined EMG and vision data in a gesture classification task, and Taunyazov et al [40] combined vision and tactile data in a grasping task.…”

Section: B Direct Deep Snn Trainingmentioning

confidence: 99%

“…Much work remains to be done on the spiking algorithm front, but steady progress is being made. Early efforts demonstrate digit recognition, fusion of visual and tactile perception [40], fusion of visual and EMG information [39], persistent attention and tracking [68], and online learning of gestures [102] using event-based sensors interfaced to Loihi.…”

Section: A Event-based Sensing and Perceptionmentioning

confidence: 99%

See 1 more Smart Citation

Advancing Neuromorphic Computing With Loihi: A Survey of Results and Outlook

et al. 2021

View full text Add to dashboard Cite

Deep artificial neural networks apply principles of the brain's information processing that led to breakthroughs in machine learning spanning many problem domains. Neuromorphic computing aims to take this a step further to chips more directly inspired by the form and function of biological neural circuits, so they can process new knowledge, adapt, behave, and learn in real time at low power levels. Despite several decades of research, until recently, very few published results have shown that today's neuromorphic chips can demonstrate quantitative computational value. This is now changing with the advent of Intel's Loihi, a neuromorphic research processor designed to support a broad range of spiking neural networks with sufficient scale, performance, and features to deliver competitive results compared to state-of-the-art contemporary computing architectures. This survey reviews results that are obtained to date with Loihi across the major algorithmic domains under study, including deep learning approaches and novel approaches that aim to more directly harness the key features of spike-based neuromorphic hardware. While conventional feedforward deep neural networks show modest if any benefit on Loihi, more brain-inspired networks using recurrence, precise spike-timing relationships, synaptic plasticity, stochasticity, and sparsity perform certain computation with orders of magnitude lower latency and energy compared to state-of-the-art conventional approaches. These compelling neuromorphic networks solve a diverse range of problems representative of brain-like computation, such as event-based

show abstract

Section: B Direct Deep Snn Trainingmentioning

confidence: 99%

Section: A Event-based Sensing and Perceptionmentioning

confidence: 99%

Advancing Neuromorphic Computing With Loihi: A Survey of Results and Outlook

et al. 2021

View full text Add to dashboard Cite

show abstract

“…A few other works tried to embed multiple camera sensors inside the tactile sensor to retrieve the best possible internal tactile force fields [ 22 ]. On the other hand, there has been an appeal and enthusiasm towards the learning-based approaches inculcating deep learning for the estimation of tactile information [ 23 ]. The visual-based tactile sensing mechanism can be typically classified into two approaches, such as traditional image processing/computer vision-based methods and learning-based methods.…”

Section: Literature Reviewmentioning

confidence: 99%

Vision-Based Tactile Sensor Mechanism for the Estimation of Contact Position and Force Distribution Using Deep Learning

Kakani

Cui

Ma³

et al. 2021

Sensors

View full text Add to dashboard Cite

This work describes the development of a vision-based tactile sensor system that utilizes the image-based information of the tactile sensor in conjunction with input loads at various motions to train the neural network for the estimation of tactile contact position, area, and force distribution. The current study also addresses pragmatic aspects, such as choice of the thickness and materials for the tactile fingertips and surface tendency, etc. The overall vision-based tactile sensor equipment interacts with an actuating motion controller, force gauge, and control PC (personal computer) with a LabVIEW software on it. The image acquisition was carried out using a compact stereo camera setup mounted inside the elastic body to observe and measure the amount of deformation by the motion and input load. The vision-based tactile sensor test bench was employed to collect the output contact position, angle, and force distribution caused by various randomly considered input loads for motion in X, Y, Z directions and RxRy rotational motion. The retrieved image information, contact position, area, and force distribution from different input loads with specified 3D position and angle are utilized for deep learning. A convolutional neural network VGG-16 classification modelhas been modified to a regression network model and transfer learning was applied to suit the regression task of estimating contact position and force distribution. Several experiments were carried out using thick and thin sized tactile sensors with various shapes, such as circle, square, hexagon, for better validation of the predicted contact position, contact area, and force distribution.

show abstract

“…In the earliest work [25], we proposed a method to utilizuse an event-based camera (Dynamic Vision Sensor) in order to detect incipient slippage using traditional image processing techniques. Similarly, new approaches for slip detection with the DVS are investigated in [26][27][28]. In another work, we proposed a novel framework based on the Dynamic Vision Sensor (DVS) to acquire force magnitude and classify materials in a grasp [13].…”

Section: Related Workmentioning

confidence: 99%

Dynamic-Vision-Based Force Measurements Using Convolutional Recurrent Neural Networks

Naeini

Makris

Gan

et al. 2020

Sensors

View full text Add to dashboard Cite

In this paper, a novel dynamic Vision-Based Measurement method is proposed to measure contact force independent of the object sizes. A neuromorphic camera (Dynamic Vision Sensor) is utilizused to observe intensity changes within the silicone membrane where the object is in contact. Three deep Long Short-Term Memory neural networks combined with convolutional layers are developed and implemented to estimate the contact force from intensity changes over time. Thirty-five experiments are conducted using three objects with different sizes to validate the proposed approach. We demonstrate that the networks with memory gates are robust against variable contact sizes as the networks learn object sizes in the early stage of a grasp. Moreover, spatial and temporal features enable the sensor to estimate the contact force every 10 ms accurately. The results are promising with Mean Squared Error of less than 0.1 N for grasping and holding contact force using leave-one-out cross-validation method.

show abstract

Event-Driven Visual-Tactile Sensing and Learning for Robots

Cited by 82 publications

References 50 publications

Advancing Neuromorphic Computing With Loihi: A Survey of Results and Outlook

Advancing Neuromorphic Computing With Loihi: A Survey of Results and Outlook

Vision-Based Tactile Sensor Mechanism for the Estimation of Contact Position and Force Distribution Using Deep Learning

Dynamic-Vision-Based Force Measurements Using Convolutional Recurrent Neural Networks

Contact Info

Product

Resources

About