A Mobile Manipulation System for One-Shot Teaching of Complex Tasks in Homes

Abstract

We describe a mobile manipulation hardware and software system capable of autonomously performing complex human-level tasks in real homes, after being taught the task with a single demonstration from a person in virtual reality. This is enabled by a highly capable mobile manipulation robot, whole-body task space hybrid position/force control, teaching of parameterized primitives linked to a robust learned dense visual embeddings representation of the scene, and a task graph of the taught behaviors. We demonstrate the robustness of the approach by presenting results for performing a variety of tasks, under different environmental conditions, in multiple real homes. Our approach achieves 85% overall success rate on three tasks that consist of an average of 45 behaviors each. The video is available at: https://youtu.be/HSyAGMGikLk.

Towards Robust One-shot Task Execution using Knowledge Graph Embeddings

“…Prior work in LfD has tackled the challenging problem of extracting task plans from a single end-user demonstration [10], [11], [12], [13], [14], [15], [16], [17], [18]. These approaches present intuitive ways for end-users to program complex robot behaviors using kinesthetic teaching [10], virtual reality [11], GUI programming [12], or direct demonstration [18]. However, generalization of task plans in the case of variations between the demonstration and execution environments continues to be a challenging problem.…”

“…However, generalization of task plans in the case of variations between the demonstration and execution environments continues to be a challenging problem. Prior work has focused on task generalization by adapting action trajectories [10], composing primitive behaviors [18], or incorporating decision points into the demonstration [11]. However, these approaches rely primarily on the end-user's demonstrations to achieve robust execution.…”