Situated incremental natural language understanding using Markov Logic Networks

Kennington, Casey; Schlangen, David

doi:10.1016/j.csl.2013.06.004

Cited by 10 publications

(11 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recently, generative approaches, including our own, have been presented (Funakoshi et al, 2012;Kennington et al, 2013;Kennington et al, 2014;Kennington et al, 2015b;Engonopoulos et al, 2013) which model U as words or ngrams and the world W as a set of objects in a virtual game board, represented as a set properties or concepts (in some cases, extra-linguistic or discourse aspects were also modelled in W , such as deixis). In Matuszek et al (2014), W was represented as a distribution over properties of tangible objects and U was a Combinatory Categorical Grammar parse.…”

Section: Background: Reference Resolutionmentioning

confidence: 99%

Simple Learning and Compositional Application of Perceptually Grounded Word Meanings for Incremental Reference Resolution

Kennington¹,

Schlangen²

2015

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Confere

Self Cite

View full text Add to dashboard Cite

An elementary way of using language is to refer to objects. Often, these objects are physically present in the shared environment and reference is done via mention of perceivable properties of the objects. This is a type of language use that is modelled well neither by logical semantics nor by distributional semantics, the former focusing on inferential relations between expressed propositions, the latter on similarity relations between words or phrases. We present an account of word and phrase meaning that is perceptually grounded, trainable, compositional, and 'dialogueplausible' in that it computes meanings word-by-word. We show that the approach performs well (with an accuracy of 65% on a 1-out-of-32 reference resolution task) on direct descriptions and target/landmark descriptions, even when trained with less than 800 training examples and automatically transcribed utterances.

show abstract

Section: Background: Reference Resolutionmentioning

confidence: 99%

Simple Learning and Compositional Application of Perceptually Grounded Word Meanings for Incremental Reference Resolution

Kennington¹,

Schlangen²

2015

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Confere

Self Cite

View full text Add to dashboard Cite

show abstract

“…In such interactions, humans refer to entities in the world via definite descriptions, which makes up a major portion of human communication [29]. These references often relate to the entities that are present in the environment [22]. In task-oriented situations, these references can make up a rich set of salient objects.…”

Section: Interactive Dialogue and Knowledge Acquisitionmentioning

confidence: 99%

“…Interactive dialogue between two people is the most common site for language use [8,22]. In such interactions, humans refer to entities in the world via definite descriptions, which makes up a major portion of human communication [29].…”

Section: Interactive Dialogue and Knowledge Acquisitionmentioning

confidence: 99%

Detecting Saliency by Combining Speech and Object Detection in Indoor Environments

Thapa¹

View full text Add to dashboard Cite

“…Incremental RR has also been studied in a number of papers, including a framework for fast incremental interpretation (Schuler et al, 2009), a Bayesian filtering model approach that was sensitive to disfluencies , a model that used Markov Logic Networks to resolve objects on a screen , a model of RR and incremental feedback (Traum et al, 2012), and an approach that used a semantic representation to refer to objects (Peldszus et al, 2012;Kennington et al, 2014). However, the approaches reported there did not incorporate multi-modal information, were too slow to work in real-time, were evaluated on constrained data, or only focused on a specific type of RR, ignoring pronouns or deixis.…”

Section: Related Workmentioning

confidence: 99%

“…Following and Kennington et al (2014), we model the task at hand as one of recovering I, the intention of the speaker making the RE, where I ranges over the possible alternatives (the objects in the domain). This recovery proceeds incrementally (word by word), for RE of arbitrary length.…”

Section: The Simple Incremental Update Modelmentioning

confidence: 99%

Incrementally Tracking Reference in Human/Human Dialogue Using Linguistic and Extra-Linguistic Information

Kennington¹,

Iida²,

Tokunaga³

et al. 2015

Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

Self Cite

View full text Add to dashboard Cite

A large part of human communication involves referring to entities in the world and often these entities are objects that are visually present for the interlocutors. A system that aims to resolve such references needs to tackle a complex task: objects and their visual features need to be determined, the referring expressions must be recognised, and extra-linguistic information such as eye gaze or pointing gestures need to be incorporated. Systems that can make use of such information sources exist, but have so far only been tested under very constrained settings, such as WOz interactions. In this paper, we apply to a more complex domain a reference resolution model that works incrementally (i.e., word by word), grounds words with visually present properties of objects (such as shape and size), and can incorporate extra-linguistic information. We find that the model works well compared to previous work on the same data, despite using fewer features. We conclude that the model shows potential for use in a realtime interactive dialogue system.

show abstract

Situated incremental natural language understanding using Markov Logic Networks

Cited by 10 publications

References 24 publications

Simple Learning and Compositional Application of Perceptually Grounded Word Meanings for Incremental Reference Resolution

Simple Learning and Compositional Application of Perceptually Grounded Word Meanings for Incremental Reference Resolution

Detecting Saliency by Combining Speech and Object Detection in Indoor Environments

Incrementally Tracking Reference in Human/Human Dialogue Using Linguistic and Extra-Linguistic Information

Contact Info

Product

Resources

About