Give MEANinGS to Robots with Kitchen Clash: A VR Human Computation Serious Game for World Knowledge Accumulation

Abstract

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers. L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Model-based Prediction of Exogeneous and Endogeneous Attention Shifts During an Everyday Activity

“…While the concrete model parameters will not be transferable, the approach of task modeling through a state graph is generic and has been successfully applied in other domains [25]. We can semi-automatically identify task models for many common tasks from instructional videos [35], instruction websites [7], or serious games [26].…”

Deep understanding of everyday activity commands for household robots

“…The direction is thus prototypical of everyday activities and the directions DLU handles. In summary, this work builds upon the existing SOMA ontology [6], the semantic parser SCG [9,10], and the human computation game Kitchen Clash [34]. In this work, a pipeline is build around these three modules to parse directions into semantic specifications.…”

“…Interoperability between the modules is achieved with the help of SOMA (a). The data flow between Kitchen Clash [34] and the DLU ontology and from the DLU to the action creation is slightly unexpected: in DLU, the models derived via human computation in Kitchen Clash are stored inside the ontology. They are retrieved during the action creation to facilitate human-like behavior.…”

“…To solve the problem of precision in spatial closeness, DLU uses human computation [52]. Pfau et al collected human activity data in a virtual reality (VR) kitchen environment [34] for various tasks. 6 Inside the VR environment, users could chop vegetables, move objects around, and perform other cooking related tasks.…”