Tangled: Learning to untangle ropes with RGB-D perception

Abstract

Abstract-In this paper, we address the problem of manipulating deformable objects such as ropes. Starting with an RGB-D view of a tangled rope, our goal is to infer its knot structure and then choose appropriate manipulation actions that result in the rope getting untangled. We design appropriate features and present an inference algorithm based on particle filters to infer the rope's structure. Our learning algorithm is based on max-margin learning. We then choose an appropriate manipulation action based on the current knot structure and other properties such as slack in the rope. We then repeatedly perform perception and manipulation until the rope is untangled. We evaluate our algorithm extensively on a dataset having five different types of ropes and 10 different types of knots. We then perform robotic experiments, in which our bimanual manipulator (PR2) untangles ropes successfully 76.9% of the time.

Untangling Dense Non-Planar Knots by Learning Manipulation Features and Recovery Policies

“…One approach performs state estimation and leverages the learned state representation for downstream planning. For instance, Lui and Saxena [19] and Chi and Berenson [3] propose using classical visual feature extraction to estimate the state of deformable rope and cloth, respectively, subject to partial occlusion. Sundaresan et al [28] investigate object representation learning via dense object descriptors [24,5] for rope knot-tying and arrangement, and Ganapathi et al [6] extend this methodology to 2D fabric smoothing and folding.…”

“…To the best of our knowledge, Lui and Saxena [19] is the first published study of robot cable untying. The authors use RGB-D sensing and classical feature extraction to approximate a tangled cable as a graph consisting of cable crossings and endpoints.…”

“…The authors use RGB-D sensing and classical feature extraction to approximate a tangled cable as a graph consisting of cable crossings and endpoints. We primarily build upon the work of Grannen et al [7], which (1) presents a geometric algorithm for cable untangling based on the graphical abstraction from Lui and Saxena [19] and (2) uses deep learning to detect cable crossings with keypoint inference. These keypoints serve as input to a greedy action planner that iteratively reduces the number of crossings.…”

“…As robots automate more tasks involving 1D deformable objects, which we refer to as "cables," they will increasingly confront highly complex knots, either because the task itself requires knot untangling or because untangling cables is a prerequisite for a downstream task. Prior work has explored untangling manipulation plans consisting of highlevel manipulation primitives (e.g., pick and pull points) [28,19,7], but robots often face challenges when attempting to execute these plans in physical manipulation tasks, due to the complexity of dynamics modeling and perception in real-world settings. This motivates the need for a low-level controller which can mitigate the gap between high-level planning and physical execution.…”

“…These properties complicate the problems of both generating untangling manipulation plans and executing these plans effectively, especially in densely knotted configurations. Prior work for cable manipulation has primarily explored manipulation planning methods that learn full state information [28,19] or task-specific features [7]. These methods then execute planned actions through a sequence of analytical, hand-engineered motion primitives, which can be brittle and imprecise under increasing knot complexity and density due to layered self-occlusions, friction at cable intersections, and the need for precision in grasping.…”