Probabilistic grounding of situated speech using plan recognition and reference resolution

Gorniak, Peter; Roy, Deb

doi:10.1145/1088463.1088489

Cited by 35 publications

(15 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A related line of work focuses on grounding referring expressions to referents in 3D worlds with simple colored geometric shapes (Gorniak and Roy, 2004;Gorniak and Roy, 2005). More recent work grounds text to object attributes such as color and shape in images (Matuszek et al, 2012;Krishnamurthy and Kollar, 2013).…”

Section: Related Tasksmentioning

confidence: 99%

Text to 3D Scene Generation with Rich Lexical Grounding

Chang¹,

Monroe²,

Savva³

et al. 2015

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Confere

View full text Add to dashboard Cite

The ability to map descriptions of scenes to 3D geometric representations has many applications in areas such as art, education, and robotics. However, prior work on the text to 3D scene generation task has used manually specified object categories and language that identifies them. We introduce a dataset of 3D scenes annotated with natural language descriptions and learn from this data how to ground textual descriptions to physical objects. Our method successfully grounds a variety of lexical terms to concrete referents, and we show quantitatively that our method improves 3D scene generation over previous work using purely rule-based methods. We evaluate the fidelity and plausibility of 3D scenes generated with our grounding approach through human judgments. To ease evaluation on this task, we also introduce an automated metric that strongly correlates with human judgments.

show abstract

Section: Related Tasksmentioning

confidence: 99%

Text to 3D Scene Generation with Rich Lexical Grounding

Chang¹,

Monroe²,

Savva³

et al. 2015

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Confere

View full text Add to dashboard Cite

show abstract

“…Quickset (Cohen, et al, 1997), FUSS (Gorniak & Roy, 2005) and (Tue Vo & Wood, 1996) are examples of such direct manipulation systems in which speech is combined with a pen based interface. For virtual environments some examples are (McGlashan, 1995;Muller, et al, 1998;Cernak & Sannier, 2002;Kaiser, et al, 2003), they are usually combined with some form of direct manipulation (e.g.…”

Section: Related Workmentioning

confidence: 99%

Automatic speech grammar generation during conceptual modelling of virtual environments

Vanacken

Raymaekers

Coninx

2008

TVC

View full text Add to dashboard Cite

show abstract

“…Multiple Interpretations The particular implementation discussed here uses the best interpretation of an utterance exclusively. In previous work we have shown ways to consider multiple weighted interpretations simultaneously by probabilistically mixing the linguistic elements from the language parser with the affordances produced by the structural grammar (Gorniak & Roy, 2005a). It would clearly be beneficial to adapt those methods to the system described here to consider multiple word and constituent meanings and their interpretations simultaneously.…”

Section: Action Markersmentioning

confidence: 99%

Situated Language Understanding as Filtering Perceived Affordances

Gorniak¹,

Roy²

2007

Cognitive Science

Self Cite

View full text Add to dashboard Cite

We introduce a computational theory of situated language understanding in which the meaning of words and utterances depends on the physical environment and the goals and plans of communication partners. According to the theory, concepts that ground linguistic meaning are neither internal nor external to language users, but instead span the objective‐subjective boundary. To model the possible interactions between subject and object, the theory relies on the notion of perceived affordances: structured units of interaction that can be used for prediction at multiple levels of abstraction. Language understanding is treated as a process of filtering perceived affordances. The theory accounts for many aspects of the situated nature of human language use and provides a unified solution to a number of demands on any theory of language understanding including conceptual combination, prototypicality effects, and the generative nature of lexical items.To support the theory, we describe an implemented system that understands verbal commands situated in a virtual gaming environment. The implementation uses probabilistic hierarchical plan recognition to generate perceived affordances. The system has been evaluated on its ability to correctly interpret free‐form spontaneous verbal commands recorded from unrehearsed game play between human players. The system is able to “step into the shoes” of human players and correctly respond to a broad range of verbal commands in which linguistic meaning depends on social and physical context. We quantitatively compare the system's predictions in response to direct player commands with the actions taken by human players and show generalization to unseen data across a range of situations and verbal constructions.

show abstract

Probabilistic grounding of situated speech using plan recognition and reference resolution

Cited by 35 publications

References 17 publications

Text to 3D Scene Generation with Rich Lexical Grounding

Text to 3D Scene Generation with Rich Lexical Grounding

Automatic speech grammar generation during conceptual modelling of virtual environments

Situated Language Understanding as Filtering Perceived Affordances

Contact Info

Product

Resources

About