Proprioception and Exteroception of a Soft Robotic Finger Using Neuromorphic Vision-Based Sensing

Faris, Omar; Muthusamy, Rajkumar; Renda, Federico; Hussain, Irfan; Gan, Dongming; Seneviratne, Lakmal; Zweiri, Yahya

doi:10.1089/soro.2022.0030

Cited by 17 publications

(9 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The DelTact sensor proposed a 3D contact reconstruction method using 2D tactile images by convex optimization modeling of contact geometry and projection relationship [64]. In addition, technologies in other fields are also being migrated into the field of visuotactile sensing (e.g., neuromorphic vision-based sensing [65], [66]). We believe such attempts will provide new ideas for constructing perception mapping and further expand the applicable scenarios of 2D MDM.…”

Section: B Technologies and Implementationmentioning

confidence: 99%

Marker Displacement Method Used in Vision-Based Tactile Sensors—From 2-D to 3-D: A Review

Jiang

2023

IEEE Sensors J.

View full text Add to dashboard Cite

The vision-based tactile sensor has been proven to be a promising device for sensing tactile information. Among such sensors, the marker displacement method (MDM) is the most common method used in such sensors for representing and extracting contact information. It uses the position field and displacement field of a marker array to characterize the original tactile information, and further achieves multimodal tactile perception through original information processing. This article is the first to classify MDM into three typical categories based on the dimensionality perspective: 2D MDM, 2.5D MDM, and 3D MDM. A comparison study is presented with a focus on the principles, characteristics, applications, and distinctions of these three methods. The latest literature has also been researched as the arguments. Finally, a summary of these three categories is presented as a helpful reference.

show abstract

Section: B Technologies and Implementationmentioning

confidence: 99%

Marker Displacement Method Used in Vision-Based Tactile Sensors—From 2-D to 3-D: A Review

Jiang

2023

IEEE Sensors J.

View full text Add to dashboard Cite

show abstract

“…VBTS output images, and thus can be processed using classical and learning methods for image processing. Some algorithms build on intermediate features such as marker position, optical flow, or depth [4,5,[8][9][10][11][12][13], while others build and train end-to-end models [11,[13][14][15]. The high resolution of VBTS allows robots to perform tasks such as detecting slip [16][17][18], estimating force distribution [5,19], classifying surface textures [20,21], and manipulating of small and soft objects [22,23] as well as many tasks that require dexterity and fine detailed pose estimation.…”

Section: Of 14mentioning

confidence: 99%

“…Furthermore, the downtime between consecutive frames can contain information that is of crucial importance to precision machining. To alleviate these issues, neuromorphic event-based cameras are being employed for vision-based tactile sensing [10,15,17,[32][33][34][35][36]. Neuromorphic cameras (also known as event-based cameras) are a relatively new technology, first introduced in [37], that aims to mimic how the human eye works.…”

Section: Neuromorphic Vision-based Tactile Sensingmentioning

confidence: 99%

TactiGraph: An Asynchronous Graph Neural Network for Contact Angle Prediction Using Neuromorphic Vision-Based Tactile Sensing

Sajwani¹,

Ayyad²,

Alkendi³

et al. 2023

Preprint

View full text Add to dashboard Cite

Vision-based tactile sensors (VBTS) have become the de facto method of giving robots the ability to obtain tactile feedback from their environment. Unlike other solutions to tactile sensing, VBTS offers high spatial resolution feedback without compromising on instrumentation costs or incurring additional maintenance expenses. However, conventional cameras used in VBTS have a fixed update rate and output redundant data, leading to computational overhead downstream. In this work, we present a neuromorphic vision-based tactile sensor (N-VBTS) that employs observations from an event-based camera for contact angle prediction. Particularly, we design and develop a novel graph neural network, dubbed TactiGraph, that asynchronously operates on graphs constructed from raw N-VBTS streams exploiting their spatiotemporal correlations to perform predictions. Although conventional VBTS uses an internal illumination source, TactiGraph is reported to perform efficiently in both scenarios, with and without an internal illumination source. Rigorous experimental results revealed that TactiGraph achieved a mean absolute error of 0.62∘ in predicting the contact angle and was faster and more efficient than both conventional VBTS and other N-VBTS, with lower instrumentation costs. Specifically, N-VBTS requires only 5.5% of the compute-time needed by VBTS when both are tested on the same scenario.

show abstract

“…Amini et al integrated an Optitrack system and camera-based tactile sensing with a soft end effector in order to map surface roughness for sanding processes [17]. In Faris et al's work, proprioception and exteroception of soft fingers are enabled by tracking internal marker patterns with a neuromorphic event-based camera [18].…”

Section: B Camera-based Tactile Sensingmentioning

confidence: 99%

GelSight Fin Ray: Incorporating Tactile Sensing into a Soft Compliant Robotic Gripper

Liu¹,

Adelson²

2022

2022 IEEE 5th International Conference on Soft Robotics (RoboSoft)

View full text Add to dashboard Cite

To adapt to constantly changing environments and be safe for human interaction, robots should have compliant and soft characteristics as well as the ability to sense the world around them. Even so, the incorporation of tactile sensing into a soft compliant robot, like the Fin Ray finger, is difficult due to its deformable structure. Not only does the frame need to be modified to allow room for a vision sensor, which enables intricate tactile sensing, the robot must also retain its original mechanically compliant properties. However, adding high-resolution tactile sensors to soft fingers is difficult since many sensorized fingers, such as GelSight-based ones, are rigid and function under the assumption that changes in the sensing region are only from tactile contact and not from finger compliance. A sensorized soft robotic finger needs to be able to separate its overall proprioceptive changes from its tactile information. To this end, this paper introduces the novel design of a GelSight Fin Ray, which embodies both the ability to passively adapt to any object it grasps and the ability to perform high-resolution tactile reconstruction, object orientation estimation, and marker tracking for shear and torsional forces. Having these capabilities allow soft and compliant robots to perform more manipulation tasks that require sensing. One such task the finger is able to perform successfully is a kitchen task: wine glass reorientation and placement, which is difficult to do with external vision sensors but is easy with tactile sensing. The development of this sensing technology could also potentially be applied to other soft compliant grippers, increasing their viability in many different fields.

show abstract

Proprioception and Exteroception of a Soft Robotic Finger Using Neuromorphic Vision-Based Sensing

Cited by 17 publications

References 41 publications

Marker Displacement Method Used in Vision-Based Tactile Sensors—From 2-D to 3-D: A Review

Marker Displacement Method Used in Vision-Based Tactile Sensors—From 2-D to 3-D: A Review

TactiGraph: An Asynchronous Graph Neural Network for Contact Angle Prediction Using Neuromorphic Vision-Based Tactile Sensing

GelSight Fin Ray: Incorporating Tactile Sensing into a Soft Compliant Robotic Gripper

Contact Info

Product

Resources

About