Multi-purpose natural language understanding linked to sensorimotor experience in humanoid robots

Abstract

Humans have an amazing ability to bootstrap new knowledge. The concept of structural bootstrapping refers to mechanisms relying on prior knowledge, sensorimotor experience, and inference that can be implemented in robotic systems and employed to speed up learning and problem solving in new environments. In this context, the interplay between the symbolic encoding of the sensorimotor information, prior knowledge, planning, and natural language understanding plays a significant role. In this paper, we show how the symbolic descriptions of the world can be generated on the fly from the continuous robot's memory. We also introduce a multi-purpose natural language understanding framework that processes human spoken utterances and generates planner goals as well as symbolic descriptions of the world and human actions. Both components were tested on the humanoid robot ARMAR-III in a scenario requiring planning and plan recognition based on human-robot communication.

Cut & recombine: reuse of robot action components based on simple language instructions

“…Here geometric considerations are primarily used to adapt existing knowledge to the new situation and robotic manipulations are only considered in connection with discovered object positions. A similar question of memorizing common object locations is addressed by Ovchinnikova et al (2015). Beetz et al (2016) do indeed talk about manipulation data reuse, but for a different purpose than in our study.…”

“…Alternative studies do exist that emphasize data collection from robotic experiments in both industry-oriented (Björkelund et al, 2011; Persson et al, 2010) and service robotics domains (Beetz et al, 2016; Ovchinnikova et al, 2015; Riazuelo et al, 2015; Tenorth and Beetz, 2013; Tenorth et al, 2013; Winkler et al, 2014). In the following, we will discuss how the mentioned approaches relate to our study.…”

The ArmarX Statechart Concept: Graphical Programing of Robot Behavior

“…Since we realized a large number of robot programs for a wide variety of applications for the robots of the ARMAR series, we gained rich experience that allows us to elaborate on advantages and disadvantages of the proposed concept. The presented statechart approach has extensively been used not only to demonstrate simple tasks like the examples in this paper but also for complex skills applied in real world scenarios, including grasping, opening and closing doors, mixing, or pouring as presented in Ovchinnikova et al (2015). We think that the decision to restrict the ArmarX statecharts to a subset of Harel's original statechart definition has been shown to benefit our statechart concept, since the removed features (inter-level-transitions, history-connector) were rarely missed but improved comprehension and reusability significantly.…”

A review of methodologies for natural-language-facilitated human–robot cooperation

“…Interpreted models Fig.5. An typical interpreted model for NL instruction understanding [48]. Robot memory, real-world states and human NL instructions were integrated to instruct a robotic with plan executions.…”

“…object visual cues detected by cameras [58][59]; (4). robot sensorimotor behaviors monitored by both motion systems and computer vision systems [48]. Supported by rich information from these features, typical problems tackled in NLC include real-time communication, context-sensitive cooperation (sensor-speech alignment), machine-executable task plan generation, and implicit human request interpretation.…”