Incremental Natural Language Description of Dynamic Imagery

Abstract

Although image understanding and natural language processing constitute two major areas of AI, they have mostly been studied independently of each other. Only a few attempts have been concerned with the integration of computer vision and the generation of natural language expressions for the description of image sequences.The aim of our joint efforts at combining a vision system and a natural language access system is the automatic simultaneous description of dynamic imagery, i.e., we are interested in image interpretation and language processing on an incremental basis. In this contribution 1 we sketch an approach towards the integration of the Karlsruhe vision system called Actions and the natural language component Vitra developed in Saarbrücken. The steps toward realization, based 1 The work described here was partly supported by the Sonderforschungsbereich 314 der Deutschen Forschungsgemeinschaft, "Künstliche Intelligenz und wissensbasierte Systeme", projects V1 (IITB, Karlsruhe) and N2: VITRA (Universität des Saarlandes).

Plan-based integration of natural language and graphics generation

“…[65]) and VITRA systems (cf. [1,27]), which automatically create natural language descriptions of pictures and image sequences shown on the screen. These projects resulted in a better understanding of how perception interacts with language production.…”

“…[27], for details of VITRA's animation component see [56]), the AnimNL project (cf. [10]) and recent extensions of COMET (cf.…”

Multimodality in Language and Speech Systems

“…The system called REPLAI-II takes observations from image sequences as input. Moving objects from two-dimensional image sequences have been extracted by a vision system (Herzog et al, 1989) and spatia-temporal entities (spatial relations and events) have been recognised by an event-recognition system. A focussing process selects interesting agents to be concentrated on during a planrecognition process.…”

“…Each recognised intentional entity is described in natural language. A system called SOCCER (Andre et al, 1988;Herzog et al, 1989) verbalizes real-world image sequences of soccer games in natural language and REPLAI-II extends the range of capabilities of SOCCER. Here, NLP is used more for annotation through text generation with less focus on analysis.…”

VITRA: Verbalisierung visueller Information

“…-Beantwortung von Fragenüber Beobachtungen in einer Straßenverkehrsszene [3], [37] -Generierung von Simultanbeschreibungen anhand kurzer Ausschnitte aus Videoaufnahmen eines Fußballspiels [4], [18] sowie verschiedener Straßenverkehrsszenen mit Fahrzeugen und Fußgängern [17] -Erzeugung inkrementeller, multimodaler Wegbeschreibungen basierend auf einem 3D-Modell des Saarbrücker Campus [15], [28] -Natürlichsprachliche Interaktion mit einem autonomen mobilen Robotersystem [27], [40] Der vorliegende Beitrag konzentriert sich auf die automatische Auswertung und Beschreibung zeitveränderlicher Szenen ausgehend von Kamerabildfolgen.…”