Clear Grasp: 3D Shape Estimation of Transparent Objects for Manipulation

Sajjan, Shreeyak S.; Moore, Matthew; Pan, Mike; Nagaraja, Ganesh; Lee, Johnny; Zeng, Andy; Song, Shuran

doi:10.1109/icra40945.2020.9197518

Cited by 183 publications

(236 citation statements)

References 39 publications

Supporting

Mentioning

207

Contrasting

Order By: Relevance

“…7 Examples of object (scene) segmentation: (left) a 3D input pointcloud; (right) a network prediction; The colors of the points represent the object labels (adapted from [26]) commonplace in every household, this is an issue that surely needs to be addressed. S. Sajjan et al, [171] took a step towards addressing such limitations for object grasping and manipulation. In particular, they proposed a deep learning approach, named ClearGrasp [171], for estimating accurate 3D geometry of transparent objects from a single RGB-D image (which is a combination of an RBG image and a corresponding depth image) for robotic manipulation.…”

Section: Object Detectionmentioning

confidence: 99%

“…S. Sajjan et al, [171] took a step towards addressing such limitations for object grasping and manipulation. In particular, they proposed a deep learning approach, named ClearGrasp [171], for estimating accurate 3D geometry of transparent objects from a single RGB-D image (which is a combination of an RBG image and a corresponding depth image) for robotic manipulation.…”

Section: Object Detectionmentioning

confidence: 99%

See 1 more Smart Citation

The State of Lifelong Learning in Service Robots:

Kasaei

Melsen

Beers

et al. 2021

J Intell Robot Syst

View full text Add to dashboard Cite

Service robots are appearing more and more in our daily life. The development of service robots combines multiple fields of research, from object perception to object manipulation. The state-of-the-art continues to improve to make a proper coupling between object perception and manipulation. This coupling is necessary for service robots not only to perform various tasks in a reasonable amount of time but also to continually adapt to new environments and safely interact with non-expert human users. Nowadays, robots are able to recognize various objects, and quickly plan a collision-free trajectory to grasp a target object in predefined settings. Besides, in most of the cases, there is a reliance on large amounts of training data. Therefore, the knowledge of such robots is fixed after the training phase, and any changes in the environment require complicated, time-consuming, and expensive robot re-programming by human experts. Therefore, these approaches are still too rigid for real-life applications in unstructured environments, where a significant portion of the environment is unknown and cannot be directly sensed or controlled. In such environments, no matter how extensive the training data used for batch learning, a robot will always face new objects. Therefore, apart from batch learning, the robot should be able to continually learn about new object categories and grasp affordances from very few training examples on-site. Moreover, apart from robot self-learning, non-expert users could interactively guide the process of experience acquisition by teaching new concepts, or by correcting insufficient or erroneous concepts. In this way, the robot will constantly learn how to help humans in everyday tasks by gaining more and more experiences without the need for re-programming. In this paper, we review a set of previously published works and discuss advances in service robots from object perception to complex object manipulation and shed light on the current challenges and bottlenecks.

show abstract

Section: Object Detectionmentioning

confidence: 99%

Section: Object Detectionmentioning

confidence: 99%

The State of Lifelong Learning in Service Robots:

Kasaei

Melsen

Beers

et al. 2021

J Intell Robot Syst

View full text Add to dashboard Cite

show abstract

“…Visual Perception: While the RGBD-based 6D pose tracker is robust to a variety of objects with challenging properties such as textureless, reflective, geometrical featureless, or thin/small shapes, it struggles to track severely shiny, glossy, or transparent objects, due to the degenerated depth sensing of the camera. In future work, we hope to explore extending this framework to these other types of scenarios with the techniques of depth enhancement and completion [64]. In addition, the current framework requires an object CAD model beforehand to perform 6D pose tracking and for reasoning about the task of peg-in-hole insertion.…”

Section: System Observations and Limitationsmentioning

confidence: 99%

Vision-driven Compliant Manipulation for Reliable; High-Precision Assembly Tasks

Morgan¹,

Wen²,

Liang³

et al. 2021

Robotics: Science and Systems XVII

View full text Add to dashboard Cite

Highly constrained manipulation tasks continue to be challenging for autonomous robots as they require high levels of precision, typically less than 1mm, which is often incompatible with what can be achieved by traditional perception systems. This paper demonstrates that the combination of state-of-the-art object tracking with passively adaptive mechanical hardware can be leveraged to complete precision manipulation tasks with tight, industrially-relevant tolerances (0.25mm). The proposed control method closes the loop through vision by tracking the relative 6D pose of objects in the relevant workspace. It adjusts the control reference of both the compliant manipulator and the hand to complete object insertion tasks via within-hand manipulation. Contrary to previous efforts for insertion, our method does not require expensive force sensors, precision manipulators, or timeconsuming, online learning, which is data hungry. Instead, this effort leverages mechanical compliance and utilizes an objectagnostic manipulation model of the hand learned offline, offthe-shelf motion planning, and an RGBD-based object tracker trained solely with synthetic data. These features allow the proposed system to easily generalize and transfer to new tasks and environments. This paper describes in detail the system components and showcases its efficacy with extensive experiments involving tight tolerance peg-in-hole insertion tasks of various geometries as well as open-world constrained placement tasks.

show abstract

“…We transfer pre-trained weights from these tasks to learn suction and grasping affordance models, then report picking success rates while fine-tuning with trial and error on the training objects in Figure 7. We also execute test runs with novel unseen objects (including transparent objects [37]) using the affordance models and report their performance in Figure 7. These results highlight three key observations: 1) we validate that our findings in simulation apply to our real-world setting, 2) off-the-shelf vision models generalize better than vision models trained in the target environment (COCO-fix vs. Random on unseen objects, Figure 7b,d) 3) models initialized with COCO features perform better than models initialized with ImageNet features.…”

Section: B Real-world Experimentsmentioning

confidence: 99%

Learning to See before Learning to Act: Visual Pre-training for Manipulation

Lin

Zeng

Song

et al. 2020

2020 IEEE International Conference on Robotics and Automation (ICRA)

Self Cite

View full text Add to dashboard Cite

Does having visual priors (e.g. the ability to detect objects) facilitate learning to perform vision-based manipulation (e.g. picking up objects)? We study this problem under the framework of transfer learning, where the model is first trained on a passive vision task (i.e., the data distribution does not depend on the agent's decisions), then adapted to perform an active manipulation task (i.e., the data distribution does depend on the agent's decisions). We find that pre-training on vision tasks significantly improves generalization and sample efficiency for learning to manipulate objects. However, realizing these gains requires careful selection of which parts of the model to transfer. Our key insight is that outputs of standard vision models highly correlate with affordance maps commonly used in manipulation. Therefore, we explore directly transferring model parameters from vision networks to affordance prediction networks, and show that this can result in successful zeroshot adaptation, where a robot can pick up certain objects with zero robotic experience. With just a small amount of robotic experience, we can further fine-tune the affordance model to achieve better results. With just 10 minutes of suction experience or 1 hour of grasping experience, our method achieves ∼ 80% success rate at picking up novel objects.

show abstract

Clear Grasp: 3D Shape Estimation of Transparent Objects for Manipulation

Cited by 183 publications

References 39 publications

The State of Lifelong Learning in Service Robots:

The State of Lifelong Learning in Service Robots:

Vision-driven Compliant Manipulation for Reliable; High-Precision Assembly Tasks

Learning to See before Learning to Act: Visual Pre-training for Manipulation

Contact Info

Product

Resources

About