Goal-directed robot manipulation through axiomatic scene estimation

Abstract

Performing robust goal-directed manipulation tasks remains a crucial challenge for autonomous robots. In an ideal case, shared autonomous control of manipulators would allow human users to specify their intent as a goal state and have the robot reason over the actions and motions to achieve this goal. However, realizing this goal remains elusive due to the problem of perceiving the robot’s environment. We address and describe the problem of axiomatic scene estimation for robot manipulation in cluttered scenes which is the estimation of a tree-structured scene graph describing the configuration of objects observed from robot sensing. We propose generative approaches to scene inference (as the axiomatic particle filter, and the axiomatic scene estimation by Markov chain Monte Carlo based sampler) of the robot’s environment as a scene graph. The result from AxScEs estimation are axioms amenable to goal-directed manipulation through symbolic inference for task planning and collision-free motion planning and execution. We demonstrate the results for goal-directed manipulation of multi-object scenes by a PR2 robot.

Semantic Robot Programming for Goal-Directed Manipulation in Cluttered Scenes

“…Liu et al [23] also estimate a scene graph given observations, however, their approach approximates objects as oriented bounding boxes. Sui et al proposed a generative approach (AxMC) [35] for scene graph estimation and use Markov Chain Monte Carlo (MCMC) to search for the best scene graph hypothesis that explains the observations. Both D2P and AxMC assume that the robot knows what objects are present in the scene, and objects are standing in their upright poses, thus both methods can only estimate 3 DOF poses of objects (i.e., x, y, θ ).…”

“…Towards natural and intuitive modes of human-robot communication, we present the Semantic Robot Programming (SRP) paradigm for declarative robot programming over user demonstrated scenes. In SRP, we assume a robot is capable of goal-directed manipulation [35] for realizing an arbitrary scene state in the world. A user can program such goal-directed robots by demonstrating their desired goal scene.…”

“…In our approach to this problem, we must bridge the gap of interoperation between semantic mapping and existing methods for goal-directed task planning [12], [21], grasp planning [37] and motion planning [33]. We have previously proposed methods for scene estimation [34], [35] from robot RGBD sensing that used scene graphs expressed axiomatically as a semantic mapping abstraction. This abstraction allowed for ready use with modern task, grasp, and motion planning systems.…”

Semantic Mapping with Simultaneous Object Detection and Localization

“…Zhen et al [43] generated scene hypotheses based on object detections returned by R-CNN [10], and they used Bayesian based bootstrap filter to estimate object poses. Similarly, Sui et al [35] and Narayanan et al [24] proposed generative approach for object pose estimation given RGB-D observation. Discriminative object pose estimation methods use local [14], [28] or global [29], [1] descriptors to estimate object poses via feature matching.…”

SUM: Sequential scene understanding and manipulation

“…Previously, we addressed the problem of perception for goal-directed manipulation as axiomatic scene estimation [32], [33], sharing similar aims to existing work in scene estimation for manipulation of rigid objects [22], [23], [19], [17], [6]. These methods take a generative multi-hypothesis approach to robustly inferring a tree-structured scene graph, as object poses and directed inter-object relations, from cluttered scenes observed as 3D point clouds.…”