Real-time Soft Body 3D Proprioception via Deep Vision-based Sensing

Wang, Ruoyu; Wang, Shiheng; Du, Songyu; Xiao, Erdong; Yuan, Wenzhen; Chen, Feng

doi:10.48550/arxiv.1904.03820

Cited by 2 publications

(4 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Embedded cameras will be able to provide rich and powerful high-resolution information which is highly synergistic with recent image-based deep learning techniques [15]. Wang et al [9] explored the usage of embedded cameras for sensing the shape of soft robots. However, their group focused on perception and the soft robots that they studied cannot perform grasping or manipulation tasks because the camera is installed at the base of the robot.…”

Section: Intraphalangial Camerasmentioning

confidence: 99%

“…On the other hand, vision-based sensors can generate powerful and rich sensing information which potentially correspond well to the high dimensionality of deformation in soft robots. Wang et al [9] started to introduce embedded vision sensors for soft robot proprioception. They proposed a framework to predict real-time 3D shapes of robots using a neural network.…”

Section: Related Work a Soft Robot Proprioceptionmentioning

confidence: 99%

“…own state (proprioception) in addition to external stimuli (exteroception) to perform tasks as smoothly as biological systems [6]. However, due to the high flexibility of soft robots, soft robot proprioception and exteroception remain considerable challenges that significantly limit their applications [3], [6], [9].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Exoskeleton-covered soft finger with vision-based proprioception and tactile sensing

She

Liu

et al. 2019

Preprint

View full text Add to dashboard Cite

Soft robots offer significant advantages in adaptability, safety, and dexterity compared to conventional rigidbody robots. However, it is challenging to equip soft robots with accurate proprioception and exteroception due to their high flexibility and elasticity. In this work, we develop a novel exoskeleton-covered soft finger with embedded cameras and deep learning methods that enable high-resolution proprioceptive sensing and rich tactile sensing. To do so, we design features along the axial direction of the finger, which enable highresolution proprioceptive sensing, and incorporate a reflective ink coating on the surface of the finger to enable rich tactile sensing. We design a highly underactuated exoskeleton with a tendon-driven mechanism to actuate the finger. Finally, we assemble 2 of the fingers together to form a robotic gripper and successfully perform a bar stock classification task, which requires both shape and tactile information. We train neural networks for proprioception and shape (box versus cylinder) classification using data from the embedded sensors. The proprioception CNN had over 99% accuracy on our testing set (all six joint angles were within 1 • of error) and had an average accumulative distance error of 0.77 mm during live testing, which is better than human finger proprioception. These proposed techniques offer soft robots the high-level ability to simultaneously perceive their proprioceptive state and peripheral environment, providing potential solutions for soft robots to solve everyday manipulation tasks. We believe the methods developed in this work can be widely applied to different designs and applications.

show abstract

Section: Intraphalangial Camerasmentioning

confidence: 99%

Section: Related Work a Soft Robot Proprioceptionmentioning

confidence: 99%

See 1 more Smart Citation

Exoskeleton-covered soft finger with vision-based proprioception and tactile sensing

She

Liu

et al. 2019

Preprint

View full text Add to dashboard Cite

show abstract

“…The geometric response of the soft material provides a versatile source of information that captures the underlying dynamics during physical interaction [4], [12], [13]. The current development of visual sensors provides a robust mechanism for capturing the geometric deformations of the soft matter [14].…”

Section: Introductionmentioning

confidence: 99%

Scalable Tactile Sensing for an Omni-adaptive Soft Robot Finger

Yang,

Ge,

Wan

et al. 2020

Preprint

View full text Add to dashboard Cite

Robotic fingers made of soft material and compliant structures usually lead to superior adaptation when interacting with the unstructured physical environment. In this paper, we present an embedded sensing solution using optical fibers for an omni-adaptive soft robotic finger with exceptional adaptation in all directions. In particular, we managed to insert a pair of optical fibers inside the finger's structural cavity without interfering with its adaptive performance. The resultant integration is scalable as a versatile, low-cost, and moistureproof solution for physically safe human-robot interaction. In addition, we experimented with our finger design for an object sorting task and identified sectional diameters of 94% objects within the ±6mm error and measured 80% of the structural strains within ±0.1mm/mm error. The proposed sensor design opens many doors in future applications of soft robotics for scalable and adaptive physical interactions in the unstructured environment.

show abstract

Real-time Soft Body 3D Proprioception via Deep Vision-based Sensing

Cited by 2 publications

References 31 publications

Exoskeleton-covered soft finger with vision-based proprioception and tactile sensing

Exoskeleton-covered soft finger with vision-based proprioception and tactile sensing

Scalable Tactile Sensing for an Omni-adaptive Soft Robot Finger

Contact Info

Product

Resources

About