Situated open world reference resolution for human-robot dialogue

Williams, Tom; Acharya, Saurav; Schreitter, Stephanie; Scheutz, Matthias

doi:10.1109/hri.2016.7451767

Cited by 25 publications

(17 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Referring expression recognition and generation is a well studied problem in intelligent user interfaces (Chai et al, 2004), human-robot interaction (Fang et al, 2012;Chai et al, 2014;Williams et al, 2016), and situated dialogue (Kennington and Schlangen, 2017). Kazemzadeh et al (2014) and Mao et al (2016) introduce two benchmark datasets for referring expression recognition.…”

Section: Related Workmentioning

confidence: 99%

Visual Referring Expression Recognition: What Do Systems Actually Learn?

Cirik¹,

Morency²,

Berg-Kirkpatrick³

2018

Proceedings of the 2018 Conference of the North American Chapter Of the Association for Computational Linguistics: Hu

View full text Add to dashboard Cite

We present an empirical analysis of state-ofthe-art systems for referring expression recognition -the task of identifying the object in an image referred to by a natural language expression -with the goal of gaining insight into how these systems reason about language and vision. Surprisingly, we find strong evidence that even sophisticated and linguistically-motivated models for this task may ignore linguistic structure, instead relying on shallow correlations introduced by unintended biases in the data selection and annotation process. For example, we show that a system trained and tested on the input image without the input referring expression can achieve a precision of 71.2% in top-2 predictions. Furthermore, a system that predicts only the object category given the input can achieve a precision of 84.2% in top-2 predictions. These surprisingly positive results for what should be deficient prediction scenarios suggest that careful analysis of what our models are learning -and further, how our data is constructed -is critical as we seek to make substantive progress on grounded language tasks.

show abstract

Section: Related Workmentioning

confidence: 99%

Visual Referring Expression Recognition: What Do Systems Actually Learn?

Cirik¹,

Morency²,

Berg-Kirkpatrick³

2018

Proceedings of the 2018 Conference of the North American Chapter Of the Association for Computational Linguistics: Hu

View full text Add to dashboard Cite

show abstract

“…As shown in Figure 1, the clarification process ultimately involves a large number of architectural components. Our proposed module interacts directly with the architectural components for reference resolution [52,55], pragmatic generation [8,53,59], and dialogue, belief, and goal management [5,6,38,39].…”

Section: Architectural Integrationmentioning

confidence: 99%

Enabling Morally Sensitive Robotic Clarification Requests

Jackson

Williams

2022

J. Hum.-Robot Interact.

Self Cite

View full text Add to dashboard Cite

The design of current natural language-oriented robot architectures enables certain architectural components to circumvent moral reasoning capabilities. One example of this is reflexive generation of clarification requests as soon as referential ambiguity is detected in a human utterance. As shown in previous research, this can lead robots to (1) miscommunicate their moral dispositions and (2) weaken human perception or application of moral norms within their current context. We present a solution to these problems by performing moral reasoning on each potential disambiguation of an ambiguous human utterance and responding accordingly, rather than immediately and naively requesting clarification. We implement our solution in the Distributed Integrated Cognition Affect and Reflection robot architecture, which, to our knowledge, is the only current robot architecture with both moral reasoning and clarification request generation capabilities. We then evaluate our method with a human subjects experiment, the results of which indicate that our approach successfully ameliorates the two identified concerns.

show abstract

“…Probabilistic models are another popular approach [12,14,45]. While effective at identifying referents from a set of entities, they may lead to dispreferred questions, compared to approaches that leverage features or knowledge about the environment [47].…”

Section: Situated Reference Resolutionmentioning

confidence: 99%

Decision-Theoretic Question Generation for Situated Reference Resolution: An Empirical Study and Computational Model

Gervits

Briggs

Roque

et al. 2021

Proceedings of the 2021 International Conference on Multimodal Interaction

Self Cite

View full text Add to dashboard Cite

Dialogue agents that interact with humans in situated environments need to manage referential ambiguity across multiple modalities and ask for help as needed. However, it is not clear what kinds of questions such agents should ask nor how the answers to such questions can be used to resolve ambiguity. To address this, we analyzed dialogue data from an interactive study in which participants controlled a virtual robot tasked with organizing a set of tools while engaging in dialogue with a live, remote experimenter. We discovered a number of novel results, including the distribution of question types used to resolve ambiguity and the influence of dialogue-level factors on the reference resolution process. Based on these empirical findings we: (1) developed a computational model for clarification requests using a decision network with an entropybased utility assignment method that operates across modalities, (2) evaluated the model, showing that it outperforms a slot-filling baseline in environments of varying ambiguity, and (3) interpreted the results to offer insight into the ways that agents can ask questions to facilitate situated reference resolution. CCS CONCEPTS• Human-centered computing → Empirical studies in HCI; • Computing methodologies → Discourse, dialogue and pragmatics; Bayesian network models.

show abstract

Situated open world reference resolution for human-robot dialogue

Cited by 25 publications

References 17 publications

Visual Referring Expression Recognition: What Do Systems Actually Learn?

Visual Referring Expression Recognition: What Do Systems Actually Learn?

Enabling Morally Sensitive Robotic Clarification Requests

Decision-Theoretic Question Generation for Situated Reference Resolution: An Empirical Study and Computational Model

Contact Info

Product

Resources

About