Task-Driven Perception and Manipulation for Constrained Placement of Unknown Objects

Mitash, Chaitanya; Shome, Rahul; Wen, Bowen; Boularias, Abdeslam; Bekris, Kostas E.

doi:10.1109/lra.2020.3006816

Cited by 40 publications

(29 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Later, Gao and Tedrake augmented this with shape completion, which was useful for avoiding collisions when planning arm motions with the held object [17]. Finally, Mitash et al addressed the problem by fusing multiple sensor views and allowing a single regrasp as necessary, conservatively assuming the object is as large as its unobserved region [18]. None of these considered multiple regrasps or compared different ways of explicitly accounting for segmentation and completion uncertainty, as we do here.…”

Section: B Pick-place Of Novel Objectsmentioning

confidence: 99%

Robotic Pick-and-Place With Uncertain Object Instance Segmentation and Shape Completion

Gualtieri

Platt

2021

IEEE Robot. Autom. Lett.

View full text Add to dashboard Cite

In this paper we consider joint perception and control of a pick-place system. It is important to consider perception and control jointly as some actions are more likely to succeed than others given non-uniform, perceptual uncertainty. Our approach is to combine 3D object instance segmentation and shape completion with classical regrasp planning. We use the perceptual modules to estimate their own uncertainty and then incorporate this uncertainty as a regrasp planning cost. We compare 7 different regrasp planning cost functions, 4 of which explicitly model probability of plan execution success. Results show uncertainty-aware costs improve performance for complex tasks, e.g., for a bin packing task, object placement success is 6.2% higher in simulation and 4.1% higher in the real world with an uncertainty-aware cost versus the commonly used minimum-number-of-grasps cost.

show abstract

Section: B Pick-place Of Novel Objectsmentioning

confidence: 99%

Robotic Pick-and-Place With Uncertain Object Instance Segmentation and Shape Completion

Gualtieri

Platt

2021

IEEE Robot. Autom. Lett.

View full text Add to dashboard Cite

show abstract

“…In particular, [48] developed a compliant manipulation system by integrating a 6D object pose tracker and a reactive motion planner. To generalize to objects of unknown shapes, [49] presents a system that maintains a dual representation of the unseen object's shape through visual tracking, achieving constrained placement. Although promising results of manipulating one or a few objects have been shown, their scalability to other objects or precision placement tasks remains unclear.…”

Section: Related Workmentioning

confidence: 99%

“…In addition, the current framework requires an object CAD model beforehand to perform 6D pose tracking and for reasoning about the task of peg-in-hole insertion. In future work, reconstructing the model of novel objects on-the-fly [49] while with sufficient precision to perform high-tolerance tight insertion tasks is of interest.…”

Section: System Observations and Limitationsmentioning

confidence: 99%

Vision-driven Compliant Manipulation for Reliable; High-Precision Assembly Tasks

Morgan¹,

Wen²,

Liang³

et al. 2021

Robotics: Science and Systems XVII

Self Cite

View full text Add to dashboard Cite

Highly constrained manipulation tasks continue to be challenging for autonomous robots as they require high levels of precision, typically less than 1mm, which is often incompatible with what can be achieved by traditional perception systems. This paper demonstrates that the combination of state-of-the-art object tracking with passively adaptive mechanical hardware can be leveraged to complete precision manipulation tasks with tight, industrially-relevant tolerances (0.25mm). The proposed control method closes the loop through vision by tracking the relative 6D pose of objects in the relevant workspace. It adjusts the control reference of both the compliant manipulator and the hand to complete object insertion tasks via within-hand manipulation. Contrary to previous efforts for insertion, our method does not require expensive force sensors, precision manipulators, or timeconsuming, online learning, which is data hungry. Instead, this effort leverages mechanical compliance and utilizes an objectagnostic manipulation model of the hand learned offline, offthe-shelf motion planning, and an RGBD-based object tracker trained solely with synthetic data. These features allow the proposed system to easily generalize and transfer to new tasks and environments. This paper describes in detail the system components and showcases its efficacy with extensive experiments involving tight tolerance peg-in-hole insertion tasks of various geometries as well as open-world constrained placement tasks.

show abstract

“…A number of other approaches [13], [14] demonstrate systems that exploit the ability to gain information by interacting with objects. There is also a long line of work aimed at "interactive segmentation", that is, using robot motions to disambiguate among object hypotheses when manipulating in clutter [15].…”

Section: Related Workmentioning

confidence: 99%

Long-Horizon Manipulation of Unknown Objects via Task and Motion Planning with Estimated Affordances

Curtis¹,

Fang²,

Kaelbling³

et al. 2021

Preprint

View full text Add to dashboard Cite

We present a strategy for designing and building very general robot manipulation systems involving the integration of a general-purpose task-and-motion planner with engineered and learned perception modules that estimate properties and affordances of unknown objects. Such systems are closedloop policies that map from RGB images, depth images, and robot joint encoder measurements to robot joint position commands. We show that following this strategy a task-and-motion planner can be used to plan intelligent behaviors even in the absence of a priori knowledge regarding the set of manipulable objects, their geometries, and their affordances. We explore several different ways of implementing such perceptual modules for segmentation, property detection, shape estimation, and grasp generation. We show how these modules are integrated within the PDDLStream task and motion planning framework. Finally, we demonstrate that this strategy can enable a single system to perform a wide variety of real-world multi-step manipulation tasks, generalizing over a broad class of objects, object arrangements, and goals, without any prior knowledge of the environment and without re-training.

show abstract

Task-Driven Perception and Manipulation for Constrained Placement of Unknown Objects

Cited by 40 publications

References 25 publications

Robotic Pick-and-Place With Uncertain Object Instance Segmentation and Shape Completion

Robotic Pick-and-Place With Uncertain Object Instance Segmentation and Shape Completion

Vision-driven Compliant Manipulation for Reliable; High-Precision Assembly Tasks

Long-Horizon Manipulation of Unknown Objects via Task and Motion Planning with Estimated Affordances

Contact Info

Product

Resources

About