Training a Multilingual Sportscaster: Using Perceptual Context to Learn Language

Abstract

We present a novel framework for learning to interpret and generate language using only perceptual context as supervision. We demonstrate its capabilities by developing a system that learns to sportscast simulated robot soccer games in both English and Korean without any language-specific prior knowledge. Training employs only ambiguous supervision consisting of a stream of descriptive textual comments and a sequence of events extracted from the simulation trace. The system simultaneously establishes correspondences between individual comments and the events that they describe while building a translation model that supports both parsing and generation. We also present a novel algorithm for learning which events are worth describing. Human evaluations of the generated commentaries indicate they are of reasonable quality and in some cases even on par with those produced by humans for our limited domain.

How to talk to strangers: Generating medical reports for first-time users

“…a) Natural Language Generation:: Natural Language Generation from time-series data has been investigated for various tasks, including, but not limited to, weather forecast generation [9], [10], [11], sportcasting [12], [13], [14], narrative generation to assist children with communication needs [15], student feedback generation [16]and report generation from students data [17] and report generation from medical time-series data [18] [19][20] [4].…”

A First-Order-Logic Based Model for Grounded Language Learning

“…In seminal work on grounded language learning, Siskind [19] focused on learning word meaning, but not grammatical structure. In a series of work, Mooney and colleagues [9,4,3,11] learn to match phrases to elements of a context. Their goals are similar to ours, but an essential difference is that they only map a phrase to a single element in the context; that is, the meaning of the phrase must be a single element of the context.…”

“…Their goals are similar to ours, but an essential difference is that they only map a phrase to a single element in the context; that is, the meaning of the phrase must be a single element of the context. Chen et al [3] acknowledge this limitation, and mention inductive logic programming (ILP) as a possible approach to learning more complex meanings, to be explored in future work. This work partially fills that gap.…”

“…We present a learning algorithm that analyzes utterances and the context in which they are produced to create a language model that can be used to map sentences onto meanings and vice versa. An important difference with earlier work [9,4,3,11,13] is that our learner learns from sentence/context pairs, whereas earlier work requires the meaning, or a set of candidate meanings, to be provided for each example sentence. That is, earlier work used examples of the form (x, y), with y = f (x), or (x, Y ) with f (x) ∈ Y , to learn the function f that maps sentences to their meaning; our work uses examples of the form (x, C) with a more complex relationship between the context description C and the meaning f (x).…”

Learning Language Models from Images with ReGLL

“…Thus, some work have been done to integrate some semantic information in the language learning process. The work from Chen et al [2] is one example of this way of learning language models. Nevertheless, in this approach the meaning of each sentence has to be provided for each example of the training set.…”