The Robot as Scientist: Using Mental Simulation to Test Causal Hypotheses Extracted from Human Activities in Virtual Reality

Abstract

To act effectively in its environment, a cognitive robot needs to understand the causal dependencies of all intermediate actions leading up to its goal. For example, the system has to infer that it is instrumental to open a cupboard door before trying to grasp an object inside the cupboard. In this paper, we introduce a novel learning method for extracting instrumental dependencies by following the scientific cycle of observations, generation of causal hypotheses and testing through experiments. Our method uses a virtual reality dataset containing observations from human activities to generate hypotheses about causal dependencies between actions. It detects pairs of actions with a high temporal co-occurrence and verifies if one action is instrumental in executing the other action through mental simulation in a virtual reality environment which represents the system's mental model. Our approach is able to extract all present instrumental action dependencies while significantly reducing the search space for mental simulation, resulting in a 6-fold reduction in computational time.

Action Selection in Everyday Activities: The Opportunistic Planning Model

“…The Household Activities from Virtual Environments (HAVE) data set (Uhde et al., 2020) was recorded at the Automatica Trade Fair 2018 and consists of recordings for three scenarios, including 83 instances of table setting in a virtual environment. Each visitor could record one instance for each scenario, with each recording being limited to a maximum of 5 min.…”

“…While several approaches exist that specifically consider ill‐defined tasks (Firby, 1987; Jiménez, De La Rosa, Fernández, Fernández, & Borrajo, 2012), they are ill‐suited to explain the cognitive processes involved in human planning and action selection as they are either infeasible or inefficient in real‐world settings (Georgievski & Aiello, 2015). There are several data sets available that consider everyday activities, but without studying human action selection behavior, instead focusing on motion segmentation and action recognition (Damen, D. et al., 2018; Rohrbach et al., 2016; Rybok, Friedberger, Hanebeck, & Stiefelhagen, 2011; Tenorth, Bandouch, & Beetz, 2009), collecting biosignals of everyday activities (Meier, Mason, Porzel, Putze, & Schultz, 2018), and understanding causal dependencies of actions (Uhde, Berberich, Ramirez‐Amaro, & Cheng, 2020).…”

Why Did I Fail? A Causal-Based Method to Find Explanations for Robot Failures

“…While the main objective is to use graphical models to generalize task executions, these works don't look into the question of how these models can be utilized for failure explanations. A different letter [14] investigates the problem of learning causal relations between actions in household-related tasks. They discover, for example, that there is a causal connection between opening a drawer and retrieving plates from human demonstrations.…”

Why did I fail? A Causal-based Method to Find Explanations for Robot Failures

“…While the main objective is to use graphical models to generalize task executions, these works don't look into the question of how these models can be utilized for failure explanations. A different paper [13] investigates the problem of learning causal relations between actions in household-related tasks. They discover, for example, that there is a causal connection between opening a drawer and retrieving plates.…”