The POETICON enacted scenario corpus — A tool for human and computational experiments on action understanding

Abstract

A good data corpus lies at the heart of progress in both perceptual/cognitive science and in computer vision. While there are a few datasets that deal with simple actions, creating a realistic corpus for complex, long action sequences that contains also human-human interactions has so far not been attempted to our knowledge. Here, we introduce such a corpus for (inter)action understanding that contains six everyday scenarios taking place in a kitchen / living-room setting. Each scenario was acted out several times by different pairs of actors and contains simple object interactions as well as spoken dialogue. In addition, each scenario was first recorded with several HD cameras and also with motion-capturing of the actors and several key objects. Having access to the motion capture data allows not only for kinematic analyses, but also allows for the production of realistic animations where all aspects of the scenario can be fully controlled. We also present results from a first series of perceptual experiments that show how humans are able to infer scenario classes, as well as individual actions and objects from computer animations of everyday situations. These results can serve as a benchmark for future computational approaches that begin to take on complex action understanding.

Bridging the collectives: A review of collective human–robot construction

“…4. Situated language and the symbol grounding problem (e.g., ELIZA, 108 the Loebner Prize, 109 and the POETICON project 110 ). 5.…”

A review of verbal and non-verbal human–robot interactive communication

“…A stream of work around an approach dealing with the evolution of language and semiotics, is outlined in [87]. From a more applied and practical point of view though, one would like to be able to have grounded ontologies [88] [89] or even robotusable lexica augmented with computational models providing such grounding: and this is the ultimate goal of the EU projects POETICON [90] [91], and the follow-up project POETICON II.…”

From pose to activity: Surveying datasets and introducing CONVERSE

“…Activities include 'having a conversation', 'phone calls', 'laying down', 'drinking' and 'eating', but may also include sub-actions within a higher level task, such as 'setting a table' or 'cooking a meal'. The executions may be allowed to occur naturally as in the 50 Salads, MPII Cooking, and MPII Composite datasets; or the observations may be more scripted, such as in the POETICON and the robotic class of the TUM Kitchen set [73,16,12]. By understanding the actions and interactions within a daily activity dataset the field is moving towards learning higher level semantics of human behavior via natural representations.…”