Simple Learning and Compositional Application of Perceptually Grounded Word Meanings for Incremental Reference Resolution

Abstract

An elementary way of using language is to refer to objects. Often, these objects are physically present in the shared environment and reference is done via mention of perceivable properties of the objects. This is a type of language use that is modelled well neither by logical semantics nor by distributional semantics, the former focusing on inferential relations between expressed propositions, the latter on similarity relations between words or phrases. We present an account of word and phrase meaning that is perceptually grounded, trainable, compositional, and 'dialogueplausible' in that it computes meanings word-by-word. We show that the approach performs well (with an accuracy of 65% on a 1-out-of-32 reference resolution task) on direct descriptions and target/landmark descriptions, even when trained with less than 800 training examples and automatically transcribed utterances.

Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue

“…The main driver of the NLU in our SDS is the SIUM model of NLU introduced in Kennington et al (2013). SIUM has been used in several systems which have reported substantial results in various domains, languages, and tasks (Han et al, 2015;Kennington et al, 2015; Kennington and Schlangen, 2017) Though originally a model of reference resolution, it was always intended to be used for general NLU, which we do here. The model is formalised as follows:…”

“…The work here also builds off of Kennington and Schlangen (2015) in the same way in that their work only focused on reference to single objects. The extension of this previous work to handle more complex scene descriptions required substantial composition on the word and segment levels.…”

“…4 These classifiers are learned from instances of language use, i.e., by observing referring expressions paired with the ob-r(x,y), so x and y are jointly constrained. See Kennington and Schlangen (2015) for details on this training.…”

“…We understand individual words using the Wordsas-Classifiers (WAC) model of Kennington and Schlangen (2015). In this model, a classifier is trained for each word w p in the vocabulary.…”

Robot perception errors and human resolution strategies in situated human–robot dialogue

“…For example, integrating world-knowledge [32] and/or linguistic ontological knowledge [3]; integrating spatial semantics into a compositional/attentional accounts of reference [23,24,31]; learning spatial semantics directly from sensor data using machine learning techniques [12,34]; modelling the functional aspects of spatial semantics in terms of predicting the dynamics of objects in the scene [10,42]; capturing the vagueness and gradation of spatial semantics [17,22,43]; and leveraging analogical reasoning mechanisms to enable agents to apply spatial semantics to new environments [13].…”