Learning to Compose Hierarchical Object-Centric Controllers for Robotic Manipulation

Abstract

Manipulation tasks can often be decomposed into multiple subtasks performed in parallel, e.g., sliding an object to a goal pose while maintaining contact with a table. Individual subtasks can be achieved by task-axis controllers defined relative to the objects being manipulated, and a set of object-centric controllers can be combined in an hierarchy. In prior works, such combinations are defined manually or learned from demonstrations. By contrast, we propose using reinforcement learning to dynamically compose hierarchical object-centric controllers for manipulation tasks. Experiments in both simulation and real world show how the proposed approach leads to improved sample efficiency, zero-shot generalization to novel test environments, and simulation-to-reality transfer without fine-tuning.

Generalizing Object-Centric Task-Axes Controllers using Keypoints

“…One limitation of [4] is that they do not infer the controller parameters directly from the observed data. Instead, they use heuristics to define the set of possible controller parameterizations for each task.…”

“…Our aim in this work is to extend [4] to allow it to infer the controller parameters directly from visual input. Thus we avoid the use of fixed heuristics to find the position-target parameters for the controllers.…”

“…These methods use Imitation Learning (IL) combined with manually defined heuristics such as inter-trial variance between human demonstrations to rank proposed task-frames. Recent work [4] have also proposed using Reinforcement Learning (RL) to choose multiple different controllers both sequentially and in parallel to complete a task. Each controller in [4] is defined with respect to some task axes or target keypoint.…”

“…Recent work [4] have also proposed using Reinforcement Learning (RL) to choose multiple different controllers both sequentially and in parallel to complete a task. Each controller in [4] is defined with respect to some task axes or target keypoint. To avoid controllers at each step from interferring with each other null-space projections are employed.…”

“…Instead of using monolithic policies for task learning, recent work [4] has proposed a more modular approach by defining task-axis controllers for each possible subtask. These controllers are attached to different objects (or their parts) in the scene, such as the normal of a table or middle of the door handle.…”