Cognitive Principles in Robust Multimodal Interpretation

Chai, Joyce; Prasov, Zahar; Qu, Shaolin

doi:10.1613/jair.1936

Cited by 8 publications

(7 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Building on this work, Chai, Hong, and Zhou (2004) proposed a probabilistic graph-matching algorithm for resolving referring expressions that are complex (involving multiple target referents) and ambiguous (involving gestures that could indicate multiple candidate referents) in multimodal user interfaces. Because this algorithm had high computational complexity, Chai, Prasov, and Qu (2006) demonstrated how the algorithm's performance could be improved using a greedy algorithm based on the theories of Conversational Implicature (Dale & Reiter, 1995;Grice, 1975) and the GH. Chai et al combine these theories to create a reduced hierarchy: Gesture ⊆ Focus ⊆ Visible ⊆ Others, where Focus combines the "in focus" and "activated" tiers of the GH, and Visible combines its "familiar" and "uniquely identifiable" tiers.…”

Section: Related Workmentioning

confidence: 99%

Givenness Hierarchy Theoretic Cognitive Status Filtering

Pal¹,

Zhu²,

Golden-Lasher³

et al. 2020

Preprint

View full text Add to dashboard Cite

For language-capable interactive robots to be effectively introduced into human society, they must be able to naturally and efficiently communicate about the objects, locations, and people found in human environments. An important aspect of natural language communication is the use of pronouns. According to the linguistic theory of the Givenness Hierarchy (GH), humans use pronouns due to implicit assumptions about the cognitive statuses their referents have in the minds of their conversational partners. In previous work, Williams et al. presented the first computational implementation of the full GH for the purpose of robot language understanding, leveraging a set of rules informed by the GH literature. However, that approach was designed specifically for language understanding, oriented around GH-inspired memory structures used to assess what entities are candidate referents given a particular cognitive status. In contrast, language generation requires a model in which cognitive status can be assessed for a given entity. We present and compare two such models of cognitive status: a rule-based Finite State Machine model directly informed by the GH literature and a Cognitive Status Filter designed to more flexibly handle uncertainty. The models are demonstrated and evaluated using a silver-standard English subset of the OFAI Multimodal Task Description Corpus.

show abstract

Section: Related Workmentioning

confidence: 99%

Givenness Hierarchy Theoretic Cognitive Status Filtering

Pal¹,

Zhu²,

Golden-Lasher³

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…There has been considerable work in reference resolution, more generally, and in applying various theories from cognitive science in order to use pragmatics (Schüller 2014;Richard-Bollans, Gomez Alvarez, and Cohn 2017;Kehler 2000;Chai, Prasov, and Qu 2006;Van Deemter 2016). Unfortunately, much of this work focuses on pragmatics as they pertain to processing effort and cognitive effect on the agent, and less so on situational aspects of the agent's surroundings.…”

Section: Related Workmentioning

confidence: 99%

On Resolving Ambiguous Anaphoric Expressions in Imperative Discourse

Sarathy

Scheutz

2019

AAAI

View full text Add to dashboard Cite

Anaphora resolution is a central problem in natural language understanding. We study a subclass of this problem involving object pronouns when they are used in simple imperative sentences (e.g., “pick it up.”). Specifically, we address cases where situational and contextual information is required to interpret these pronouns. Current state-of-the art statisticallydriven coreference systems and knowledge-based reasoning systems are insufficient to address these cases. In this paper, we introduce, with examples, a general class of situated anaphora resolution problems, propose a proof-of-concept system for disambiguating situated pronouns, and discuss some general types of reasoning that might be needed.

show abstract

“…In [4] the interpretation process is defined using an edit-based transducer combined with a finitestate-based interpreter, which directly works on lattice inputs. The "referent resolution approach", used in [5], is a further approach for multimodal interpretation. It finds the most proper referents, such as specific object or objects that are the models to which the users' input has to be matched, to the referring expressions given by the user's input.…”

Section: Related Workmentioning

confidence: 99%

Ambiguity detection in multimodal systems

Caschera

Ferri

Grifoni

2008

Proceedings of the Working Conference on Advanced Visual Interfaces

View full text Add to dashboard Cite

Multimodal systems support users to communicate in a natural way according to their needs. However, the naturalness of the interaction implies that it is hard to find one and only one interpretation of the user's input. Consequently the necessity to define methods for users' input interpretation and ambiguity detection is arising. This paper proposes a theoretical approach based on a Constraint Multiset Grammar combined with Linear Logic, for representing and detecting ambiguities, and in particular semantic ambiguities, produced by the user's input. It considers user's input as a set of primitives defined as terminal elements of the grammar, composing multimodal sentences. The Linear Logic is used to define rules that allow detecting ambiguities connected to the semantics of the user's input. In particular, the paper presents the main features of the user's input and connections between the elements belonging to a multimodal sentence, and it enables to detect ambiguities that can arise during their interpretation process.

show abstract

Cognitive Principles in Robust Multimodal Interpretation

Cited by 8 publications

References 40 publications

Givenness Hierarchy Theoretic Cognitive Status Filtering

Givenness Hierarchy Theoretic Cognitive Status Filtering

On Resolving Ambiguous Anaphoric Expressions in Imperative Discourse

Ambiguity detection in multimodal systems

Contact Info

Product

Resources

About