TossingBot: Learning to Throw Arbitrary Objects With Residual Physics

Zeng, Andy; Song, Shuran; Lee, Johnny; Rodríguez, Alberto; Funkhouser, Thomas

doi:10.1109/tro.2020.2988642

Cited by 189 publications

(77 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In this case, FCNs were used to model affordance based policies. More recently Zeng et al also made use of FCNs to encode affordance based policies to learn complex behaviours such as picking and throwing objects through the interaction with the environment [ 35 ]. In spite of the fact that these FCN based models have shown to be capable to learn object affordances, they can not cope with non-euclidean data such as point clouds.…”

Section: Literature Reviewmentioning

confidence: 99%

Affordance-Based Grasping Point Detection Using Graph Convolutional Networks for Industrial Bin-Picking Applications

Iriondo

Lazkano

Ansuategi

2021

Sensors

View full text Add to dashboard Cite

Grasping point detection has traditionally been a core robotic and computer vision problem. In recent years, deep learning based methods have been widely used to predict grasping points, and have shown strong generalization capabilities under uncertainty. Particularly, approaches that aim at predicting object affordances without relying on the object identity, have obtained promising results in random bin-picking applications. However, most of them rely on RGB/RGB-D images, and it is not clear up to what extent 3D spatial information is used. Graph Convolutional Networks (GCNs) have been successfully used for object classification and scene segmentation in point clouds, and also to predict grasping points in simple laboratory experimentation. In the present proposal, we adapted the Deep Graph Convolutional Network model with the intuition that learning from n-dimensional point clouds would lead to a performance boost to predict object affordances. To the best of our knowledge, this is the first time that GCNs are applied to predict affordances for suction and gripper end effectors in an industrial bin-picking environment. Additionally, we designed a bin-picking oriented data preprocessing pipeline which contributes to ease the learning process and to create a flexible solution for any bin-picking application. To train our models, we created a highly accurate RGB-D/3D dataset which is openly available on demand. Finally, we benchmarked our method against a 2D Fully Convolutional Network based method, improving the top-1 precision score by 1.8% and 1.7% for suction and gripper respectively.

show abstract

Section: Literature Reviewmentioning

confidence: 99%

Affordance-Based Grasping Point Detection Using Graph Convolutional Networks for Industrial Bin-Picking Applications

Iriondo

Lazkano

Ansuategi

2021

Sensors

View full text Add to dashboard Cite

show abstract

“…The GSSP approach can be used for any manipulation where the actual outcomes of several practices can be used as the desired outcomes, i.e., the actual outcome can substitute the desired outcome for training. Taking throwing objects into bins as an example [46], we can practice throwing with the robot using unseen objects, record the outcome, and apply GSSP to generalize the learning model to new objects. Another example can be recording the state of food ingredients [47], or a state change [48] after the execution of a manipulation, record the action sequences, use them as training samples, and fine-tune the manipulation model to expand its generalization.…”

Section: Generalization By Self-supervised Practicingmentioning

confidence: 99%

Robot gaining accurate pouring skills through self-supervised learning and generalization

Huang

Wilches

Sun

2021

Robotics and Autonomous Systems

View full text Add to dashboard Cite

Pouring is one of the most commonly executed tasks in humans' daily lives, whose accuracy is affected by multiple factors, including the type of material to be poured and the geometry of the source and receiving containers. In this work, we propose a self-supervised learning approach that learns the pouring dynamics, pouring motion, and outcomes from unsupervised demonstrations for accurate pouring. The learned pouring model is then generalized by self-supervised practicing to different conditions such as using unaccustomed pouring cups. We have evaluated the proposed approach first with one container from the training set and four new but similar containers. The proposed approach achieved better pouring accuracy than a regular human with a similar pouring speed for all five cups. Both the accuracy and pouring speed outperform state-of-the-art works. We have also evaluated the proposed self-supervised generalization approach using unaccustomed containers that are far different from the ones in the training set. The self-supervised generalization reduces the pouring error of the unaccustomed containers to the desired accuracy level.

show abstract

“…However, the framework has only been evaluated in simulation. Zeng et al [280] have proposed end-toend formulation that jointly learns to infer control parameters for grasping and throwing motion primitives from visual observations (RGB-D images of arbitrary objects in a bin) through trial-and-error. In another work,, framework of leveraging the advantages of active perception has been presented in [281] to perform manipulation tasks.…”

Section: Vision-based Robotic Graspmentioning

confidence: 99%

Review of Deep Reinforcement Learning-Based Object Grasping: Techniques, Open Challenges, and Recommendations

2020

View full text Add to dashboard Cite

The motivation behind our work is to review and analyze the most relevant studies on deep reinforcement learning-based object manipulation. Various studies are examined through a survey of existing literature and investigation of various aspects, namely, the intended applications, techniques applied, challenges faced by researchers and recommendations for minimizing obstacles. This review refers to all relevant articles on deep reinforcement learning-based object manipulation and solutions. The object grasping issue is a major manipulation challenge. Object grasping requires detection systems, methods and tools to facilitate efficient and fast agent training. Several studies have proposed that object grasping and its subtypes are the main elements in dealing with the environment and agent. Unlike other review articles, this review article provides different observations on deep reinforcement learning-based manipulation. The results of this comprehensive review of deep reinforcement learning in the manipulation field may be valuable for researchers and practitioners because they can expedite the establishment of important guidelines.

show abstract

TossingBot: Learning to Throw Arbitrary Objects With Residual Physics

Cited by 189 publications

References 28 publications

Affordance-Based Grasping Point Detection Using Graph Convolutional Networks for Industrial Bin-Picking Applications

Affordance-Based Grasping Point Detection Using Graph Convolutional Networks for Industrial Bin-Picking Applications

Robot gaining accurate pouring skills through self-supervised learning and generalization

Review of Deep Reinforcement Learning-Based Object Grasping: Techniques, Open Challenges, and Recommendations

Contact Info

Product

Resources

About