A multimodal corpus for the evaluation of computational models for (grounded) language acquisition

Abstract

This paper describes the design and acquisition of a German multimodal corpus for the development and evaluation of computational models for (grounded) language acquisition and algorithms enabling corresponding capabilities in robots. The corpus contains parallel data from multiple speakers/actors, including speech, visual data from different perspectives and body posture data. The corpus is designed to support the development and evaluation of models learning rather complex grounded linguistic structures, e.g. syntactic patterns, from sub-symbolic input. It provides moreover a valuable resource for evaluating algorithms addressing several other learning processes, e.g. concept formation or acquisition of manipulation skills. The corpus will be made available to the public.

A Visuospatial Dataset for Naturalistic Verb Learning

“…Recent work on detecting fine-grained events in videos is particularly relevant (Hendricks et al, 2018;Zhukov et al, 2019;Fried et al, 2020, among others). Especially relevant is the data collected by Gaspers et al (2014), in which human subjects were asked to play simple games with a physical robot and narrate while doing so. Our data and work differs primarily in that we focus on the ability to ground to symbolic objects and physics rather than only to pixel data.…”