Simulation-based physics reasoning for consistent scene estimation in an HRI context

Abstract

Reasoning about spatial and geometric relations between objects in a tabletop human-robot interaction is a challenge due to the perception not being always consistent: objects placed on a table seem to be slightly in the air; they overlap; they disappear due to occlusions. Yet, interpreting and anchoring perceptual data in a physically consistent estimation of the scene is a crucial ability for humans, and thus robots in HRI context. In this paper we present a simulationbased physics reasoner integrated in a lightweight situationassessment framework called Underworlds, that allows the robot to stabilize objects and build at run-time a consistent estimation of the scene, even for entirely hidden objects, while inferring the actions performed by its human partner.

Overworld: Assessing the Geometry of the World for Human-Robot Interaction

“…Physics simulation is commonly used in robotics to predict the effects of an action [2] but not used in real-time to understand what is happening. Nevertheless, as previously explained, [22] has presented a proof of concept for HRI. In our implementation, rather than constantly simulating all the objects, we have chosen to only simulate those for which the search for an explanation is required.…”

“…In addition, the cascading architecture has the side effect to create a continuous knowledge stream triggered by each new data from each sensor. Nevertheless, it allows fast prototyping as shown in [22] where it has been used to implement simulation-based physics reasoning but not in real-time and without VPT. It is thus mainly used to represent and share multiple parallel representations of the world, to be used by other components.…”

HATP/EHDA: A Robot Task Planner Anticipating and Eliciting Human Decisions and Actions

“…inspired by [22]. Besides HATP/EHDA, the architecture embeds a situation assessment component [23] together with a semantic knowledge management [24] to assess the state of the environment and estimate human beliefs. When the supervision decides of a task to perform and if this task corresponds to a human-robot shared goal or an individual robot goal, it invokes HATP/EHDA.…”

Guiding Task through Route Description in the MuMMER Project

“…Route computing and route verbalization: The entire description of the route, from the search for the best route to get to the final destination to the verbalization of this route, is based on the Semantic Spatial Representation [6]. Geometric reasoning: It uses Underworlds [5], a lightweight framework for cascading spatio-temporal situation assessment in robotics. It represents the environment as real-time distributed data structures, containing scene graph (for representation of 3D geometries).…”