A generative probabilistic framework for learning spatial language

Abstract

The language of space and spatial relations is a rich source of abstract semantic structure. We develop a probabilistic model that learns to understand utterances that describe spatial configurations of objects in a tabletop scene by seeking the meaning that best explains the sentence chosen. The inference problem is simplified by assuming that sentences express symbolic representations of (latent) semantic relations between referents and landmarks in space, and that given these symbolic representations, utterances and physical locations are conditionally independent. As such, the inference problem factors into a symbolgrounding component (linking propositions to physical locations) and a symbol-translation component (linking propositions to parse trees). We evaluate the model by eliciting production and comprehension data from human English speakers and find that our system recovers the referent of spatial utterances at a level of proficiency approaching human performance. I. INTRODUCTIONImagine that a friend asks you to "Bring me the thing toward the far corner of the table." This simple request requires fairly sophisticated cognitive processing. You must first identify that she is referring to something on a table, in particular, one with corners. Then, you must orient the table to distinguish "far" vs. "near" corners. Finally, if there is more than one object near a "far" corner, but one is also very near the edge, you might tend to favor the other one, reasoning that it would have been easy to ask for "the one near the edge".We model a version of this problem, with a particular focus on recovering the referent of spatial utterances like "the thing toward the far corner of the table", where the only information available about the intended object is its position relative to a landmark in the scene. Beginning with no knowledge about the meanings of words, but equipped with a small vocabulary of spatial relations (e.g., containment, proximity, ordering in cardinal directions) and abstract representations of objects and their parts (e.g., a table can be represented as a line with ends and a middle, or as a rectangle with corners, quadrants and edges), our model learns probabilistic correspondences between sentences and abstract spatial relations between referents and landmarks by "observing" a teacher repeatedly generating an utterance and pointing to a location in space.Clearly a method that involved supervised learning based on observed propositional semantics would not be developmentally plausible, as children do not get to observe symbolic meaning directly, and so crucially, the abstract relations are never made overt to our learner. Rather, the locations are probabilistically assigned to abstract landmark-relation pairs

A Probabilistic Approach to Unsupervised Induction of Combinatory Categorial Grammar in Situated Human-Robot Interaction

“…Matuszek et al [25] introduced a probabilistic framework that employs categorial grammar to develop compositional representations of language and objects in the environment. Tellex et al [42] and Dawson et al [12] proposed probabilistic frameworks for grounding verbs and prepositions in utterances that encode spatial relationships between referents and landmarks. Siskind [35] developed a model for grounding semantics of verbs in short image sequences.…”

“…Noun Phrase (NP) = Determiner + Noun (N) 11. It measures homogeneity (i.e., optimal case: each cluster (separate word category) contains fewer classes of tags) and completeness (i.e., optimal case: classes of tags referring to the same cluster are equal) of clusters and classes[30] 12. It measures the variation of information of a clustering solution, so that the more the clustering is complete (i.e., high V-Measure), the lower the VI-Measure would be[26].…”

Evaluation of Word Representations in Grounding Natural Language Instructions Through Computational Human-Robot Interaction

“…In several previous studies, probabilistic models have been used for language grounding [1,10,41]. However, to the best of our knowledge, none of them included unknown synonyms and they differed in their approaches, experimental setups, or corpora from the current study, which makes the comparison of results between our study and these studies, among many others in the literature, difficult to attain.…”

“…Four different word representations have been investigated (Section III-E). The obtained F1-scores show that combining syntactic-semantic vectors and POS tags achieves the best overall grounding performance with and without the article the (Figure 8) 10 . For the Word Vector + POS Tags and Word Vector representations the model did not learn the Others modality (the article the), which might be due to the interand intra-modality distances in the employed vector space (Table III).…”

“…a word, through perceptual information [18]. Previous studies that investigated the use of crosssituational learning for grounding of objects [13,40] as well as spatial concepts [2,10,41] ensured that one word appears several times together with the same perceptual feature vector so that a corresponding mapping can be created [14]. However, natural language is ambiguous due to homonymy, i.e.…”