Learning task-oriented grasping for tool manipulation from simulated self-supervision

Fang, Kuan; Zhu, Yuke; Garg, Animesh; Kurenkov, Andrey; Mehta, Vijay; Li, Feifei; Savarese, Silvio

doi:10.1177/0278364919872545

Cited by 156 publications

(118 citation statements)

References 63 publications

Supporting

Mentioning

118

Contrasting

Order By: Relevance

“…Grasping and Manipulation in Clutter: Past approaches to this problem can be broadly characterized as model-based with geometric knowledge of the environment [3,30,41] and model-free with only raw visual input [20,24,39]. Recent studies have leveraged CNNs for casting grasping as a supervised learning problem with impressive results [11,18,19,23,25,45,49]. Pushing and singulation can facilitate grasping in cluttered scenes [5,8,17].…”

Section: Background and Related Workmentioning

confidence: 99%

Mechanical Search: Multi-Step Retrieval of a Target Object Occluded by Clutter

Danielczuk

Kurenkov

Balakrishna

et al. 2019

2019 International Conference on Robotics and Automation (ICRA)

Self Cite

112

View full text Add to dashboard Cite

When operating in unstructured environments such as warehouses, homes, and retail centers, robots are frequently required to interactively search for and retrieve specific objects from cluttered bins, shelves, or tables. Mechanical Search describes the class of tasks where the goal is to locate and extract a known target object. In this paper, we formalize Mechanical Search and study a version where distractor objects are heaped over the target object in a bin. The robot uses an RGBD perception system and control policies to iteratively select, parameterize, and perform one of 3 actions -push, suction, grasp -until the target object is extracted, or either a time limit is exceeded, or no high confidence push or grasp is available. We present a study of 5 algorithmic policies for mechanical search, with 15,000 simulated trials and 300 physical trials for heaps ranging from 10 to 20 objects. Results suggest that success can be achieved in this long-horizon task with algorithmic policies in over 95% of instances and that the number of actions required scales approximately linearly with the size of the heap. Code and supplementary material can be found at http://ai.stanford.edu/mech-search. * Authors have contributed equally and names are in alphabetical order.

show abstract

Section: Background and Related Workmentioning

confidence: 99%

Mechanical Search: Multi-Step Retrieval of a Target Object Occluded by Clutter

Danielczuk

Kurenkov

Balakrishna

et al. 2019

2019 International Conference on Robotics and Automation (ICRA)

Self Cite

112

View full text Add to dashboard Cite

show abstract

“…In addition to object attributes, task requirements dictate grasp choice for a given hand [1,30]. As such, some existing grasp planners incorporate constraints and preferences according to different manipulation tasks [31][32][33]. These methods plan grasps for different tasks, but they do not explicitly plan grasps of desired grasp types.…”

Section: Related Workmentioning

confidence: 99%

“…object class and size) and not visual features, meaning the grasp planner can only work with a limited set of known objects, unlike our work which can plan grasps for novel objects. In [33], a Task-Oriented Grasping Network (TOG-Net) is proposed to optimize a reaching and grasping policy for tool use. Similar to our approach, separate TOG-Nets are trained for each of the two tool tasks: sweeping and hammering.…”

Section: Related Workmentioning

confidence: 99%

Modeling Grasp Type Improves Learning-Based Grasp Planning

Hermans

2019

IEEE Robot. Autom. Lett.

View full text Add to dashboard Cite

Different manipulation tasks require different types of grasps. For example, holding a heavy tool like a hammer requires a multi-fingered power grasp offering stability, while holding a pen to write requires a multi-fingered precision grasp to impart dexterity on the object. In this paper, we propose a probabilistic grasp planner that explicitly models grasp type for planning high-quality precision and power grasps in real-time. We take a learning approach in order to plan grasps of different types for previously unseen objects when only partial visual information is available. Our work demonstrates the first supervised learning approach to grasp planning that can explicitly plan both power and precision grasps for a given object. Additionally, we compare our learned grasp model with a model that does not encode type and show that modeling grasp type improves the success rate of generated grasps. Furthermore we show the benefit of learning a prior over grasp configurations to improve grasp inference with a learned classifier.

show abstract

“…The comparative study of different grasping algorithms that suit research results of unknown object grasping has been presented in [271], Furthermore, an overview about recent research issues in robotic grasping and bin picking has also been presented [272], which mainly concentrated on the perception aspects of the problem, associated to computer vision algorithms. Different feasible framework based pixel input has been well studied such as adversarial learning based on ConvNet after AlexNet for effective supervise learning [273], tool use based on task-oriented grasping network (TOG-Net) [274], and grasping deformable objects based on point-level representations [275]. While learning to poke has been executed based on different algorithms such as deep neural networks for modeling the dynamics of robot's interactions directly from images [276], Self-supervised model-based approach using vision-based robotic control [277], and stochastic optimal control with latent representations (SOLAR) [278].…”

Section: Vision-based Robotic Graspmentioning

confidence: 99%

Review of Deep Reinforcement Learning-Based Object Grasping: Techniques, Open Challenges, and Recommendations

2020

View full text Add to dashboard Cite

The motivation behind our work is to review and analyze the most relevant studies on deep reinforcement learning-based object manipulation. Various studies are examined through a survey of existing literature and investigation of various aspects, namely, the intended applications, techniques applied, challenges faced by researchers and recommendations for minimizing obstacles. This review refers to all relevant articles on deep reinforcement learning-based object manipulation and solutions. The object grasping issue is a major manipulation challenge. Object grasping requires detection systems, methods and tools to facilitate efficient and fast agent training. Several studies have proposed that object grasping and its subtypes are the main elements in dealing with the environment and agent. Unlike other review articles, this review article provides different observations on deep reinforcement learning-based manipulation. The results of this comprehensive review of deep reinforcement learning in the manipulation field may be valuable for researchers and practitioners because they can expedite the establishment of important guidelines.

show abstract

Learning task-oriented grasping for tool manipulation from simulated self-supervision

Cited by 156 publications

References 63 publications

Mechanical Search: Multi-Step Retrieval of a Target Object Occluded by Clutter

Mechanical Search: Multi-Step Retrieval of a Target Object Occluded by Clutter

Modeling Grasp Type Improves Learning-Based Grasp Planning

Review of Deep Reinforcement Learning-Based Object Grasping: Techniques, Open Challenges, and Recommendations

Contact Info

Product

Resources

About