DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Reality

Handa, Ankur; Allshire, Arthur; Makoviychuk, Viktor; Petrenko, Aleksei; Singh, Ritvik; Liu, Jingzhou; Makoviichuk, Denys; Wyk, Karl Van; Alexander, Zhurkevich,; Sundaralingam, Balakumar; Narang, Yashraj

doi:10.1109/icra48891.2023.10160216

Cited by 45 publications

(10 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Nevertheless, they pose certain hypotheses about the objects and the controllers, which makes it hard to scale to more complex tasks. To overcome this limitation, deep Reinforcement Learning has been applied recently on dexterous manipulation [2,28,15,53,14,52,31]. Building on these works, incorporating demonstrations in with imitation learning also leads to better sample efficiency and more natural manipulation behaviors [55,56,4,54,72,75,38,49,3].…”

Section: Related Workmentioning

confidence: 99%

“…Building on these works, incorporating demonstrations in with imitation learning also leads to better sample efficiency and more natural manipulation behaviors [55,56,4,54,72,75,38,49,3]. However, most in-hand manipulation methods are still highly relying on visual inputs [2,28,14]. For example, Chen et al [14] propose to perform in-hand object re-orientation using depth image input, and new hardware is designed to avoid heavy occlusion.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Rotating without Seeing: Towards In-hand Dexterity through Touch

Yin,

Huang,

Qin

et al. 2023

Robotics: Science and Systems XIX

View full text Add to dashboard Cite

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Rotating without Seeing: Towards In-hand Dexterity through Touch

Yin,

Huang,

Qin

et al. 2023

Robotics: Science and Systems XIX

View full text Add to dashboard Cite

“…RL algorithms have been successfully demonstrated for inhand manipulation [9][10][11][21][22][23] tasks. One key difference between these and our work is that in in-hand manipulation, the object typically starts in close proximity to the robot, whereas in our domain, we must solve the additional problem of reaching and making contact with the object.…”

Section: B Reinforcement Learning For Contact-rich Tasksmentioning

confidence: 99%

An Evaluation of AI Integrated Education in Graduate School of Education Using the CIPP Model

Kim,

Han

2023

KAEIM

View full text Add to dashboard Cite

Fig. 1: Illustration of our tasks. The first column shows the initial robot and object poses and the desired object poses in the dark. The third column shows a subset of objects we manipulate in each domain. The middle column shows the manipulation motions. First row: the card is too flat to be grasped, so the robot must use dragging and re-orientation. Second row: the box is too large to be grasped, so the robot must push it to the bump, tumble it over, and re-orient it. Third row: the wall to the object's right is blocking all feasible grasps. The robot must first lift it up against the wall, drag it to the top, and then finally give a little push to move it to the target location. In the last two tasks, notice how the robot must both overcome and exploit the environmental contact to manipulate the object.

show abstract

“…with continuous contact or variations in friction during object interactions. In-hand manipulation employing external vision is discussed in [3], [4] and the use of Adaptive RL (reinforcement learning) policy derived from simulation torque input in robotic hands [5]. However, despite solutions to the simto-real problem, grasping objects in cluttered environments complicates policy training.…”

Section: Related Workmentioning

confidence: 99%

DenseTact: Optical Tactile Sensor for Dense Shape Reconstruction

Do¹,

Kennedy²

2022

2022 International Conference on Robotics and Automation (ICRA)

View full text Add to dashboard Cite

Collaborative robots stand to have an immense impact both on human welfare in domestic service applications and industrial superiority in advanced manufacturing requires dexterous assembly. The outstanding challenge is providing robotic fingertips with a physical design that makes them adept at performing dexterous tasks that require high-resolution, calibrated shape reconstruction and force sensing. In this work, we present DenseTact 2.0, an optical-tactile sensor capable of visualizing the deformed surface of a soft fingertip and using that image in a neural network to perform both calibrated shape reconstruction and 6-axis wrench estimation. We demonstrate the sensor accuracy of 0.3633mm per pixel for shape reconstruction, 0.410N for forces, 0.387mmN •m for torques, and the ability to calibrate new fingers through transfer learning, achieving comparable performance that trained more than four times faster with only 12% of the dataset size.

show abstract

DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Reality

Cited by 45 publications

References 12 publications

Rotating without Seeing: Towards In-hand Dexterity through Touch

Rotating without Seeing: Towards In-hand Dexterity through Touch

An Evaluation of AI Integrated Education in Graduate School of Education Using the CIPP Model

DenseTact: Optical Tactile Sensor for Dense Shape Reconstruction

Contact Info

Product

Resources

About