Learning how to Learn: An Adaptive Dialogue Agent for Incrementally
            Learning Visually Grounded Word Meanings

Yu, Yanchao; Eshghi, Arash; Lemon, Oliver

doi:10.18653/v1/w17-2802

Cited by 11 publications

(13 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Situated dialog encompasses various aspects of interaction. These include: situated Natural Language Processing (Bastianelli et al, 2016); situated reference resolution (Misu, 2018); language grounding (Johnson et al, 2017); visual question answer/visual dialog (Antol et al, 2015); dialog agents for learning visually grounded word meanings and learning from demonstration (Yu et al, 2017); and Natural Language Generation (NLG), e.g. of situated instructions and referring expressions (Byron et al, 2009;Kelleher and Kruijff, 2006).…”

Section: Related Workmentioning

confidence: 99%

Learning to Read Maps: Understanding Natural Language Instructions from Unseen Maps

Katsakioris¹,

Konstas²,

Mignotte³

et al. 2021

Proceedings of Second International Combined Workshop on Spatial Language Understanding and Grounded Communication for Robotics

View full text Add to dashboard Cite

Robust situated dialog requires the ability to process instructions based on spatial information, which may or may not be available. We propose a model, based on LXMERT, that can extract spatial information from text instructions and attend to landmarks on Open-StreetMap (OSM) referred to in a natural language instruction. Whilst, OSM is a valuable resource, as with any open-sourced data, there is noise and variation in the names referred to on the map, as well as, variation in natural language instructions, hence the need for datadriven methods over rule-based systems. This paper demonstrates that the gold GPS location can be accurately predicted from the natural language instruction and metadata with 72% accuracy for previously seen maps and 64% for unseen maps.

show abstract

Section: Related Workmentioning

confidence: 99%

Learning to Read Maps: Understanding Natural Language Instructions from Unseen Maps

Katsakioris¹,

Konstas²,

Mignotte³

et al. 2021

Proceedings of Second International Combined Workshop on Spatial Language Understanding and Grounded Communication for Robotics

View full text Add to dashboard Cite

show abstract

“…This corpus has been utilized to ground deep learning model representations of visual attributes (colors and shapes) in dialogue via interacting with a simulated tutor (Ling and Fidler, 2017;Yu et al, 2017b). Follow-up work has used this data to model a student learning names and colors of shapes using a reinforcement learning framework (Yu et al, 2016(Yu et al, , 2017a.…”

Section: Tutoring Dialogue Corpus Creationmentioning

confidence: 99%

CIMA: A Large Open Access Dialogue Dataset for Tutoring

Stasaski

Kao²,

Hearst

2020

Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications

View full text Add to dashboard Cite

One-to-one tutoring is often an effective means to help students learn, and recent experiments with neural conversation systems are promising. However, large open datasets of tutoring conversations are lacking. To remedy this, we propose a novel asynchronous method for collecting tutoring dialogue via crowdworkers that is both amenable to the needs of deep learning algorithms and reflective of pedagogical concerns. In this approach, extended conversations are obtained between crowdworkers role-playing as both students and tutors. The CIMA collection, which we make publicly available, is novel in that students are exposed to overlapping grounded concepts between exercises and multiple relevant tutoring responses are collected for the same input.CIMA contains several compelling properties from an educational perspective: student roleplayers complete exercises in fewer turns during the course of the conversation and tutor players adopt strategies that conform with some educational conversational norms, such as providing hints versus asking questions in appropriate contexts. The dataset enables a model to be trained to generate the next tutoring utterance in a conversation, conditioned on a provided action strategy.

show abstract

“…Any number of different models can be used to perform this symbol grounding, such as SVMs [59,66], Nearest Neighbor classifiers [17], and Deep Neural Models [34,62]. New concepts can also be described in terms of previously grounded wordse.g.…”

Section: Related Workmentioning

confidence: 99%

Interactive task learning via embodied corrective feedback

Appelgren

Lascarides

2020

Auton Agent Multi-Agent Syst

View full text Add to dashboard Cite

This paper addresses a task in Interactive Task Learning (Laird et al. IEEE Intell Syst 32:6–21, 2017). The agent must learn to build towers which are constrained by rules, and whenever the agent performs an action which violates a rule the teacher provides verbal corrective feedback: e.g. “No, red blocks should be on blue blocks”. The agent must learn to build rule compliant towers from these corrections and the context in which they were given. The agent is not only ignorant of the rules at the start of the learning process, but it also has a deficient domain model, which lacks the concepts in which the rules are expressed. Therefore an agent that takes advantage of the linguistic evidence must learn the denotations of neologisms and adapt its conceptualisation of the planning domain to incorporate those denotations. We show that by incorporating constraints on interpretation that are imposed by discourse coherence into the models for learning (Hobbs in On the coherence and structure of discourse, Stanford University, Stanford, 1985; Asher et al. in Logics of conversation, Cambridge University Press, Cambridge, 2003), an agent which utilizes linguistic evidence outperforms a strong baseline which does not.

show abstract

Learning how to Learn: An Adaptive Dialogue Agent for Incrementally Learning Visually Grounded Word Meanings

Cited by 11 publications

References 22 publications

Learning to Read Maps: Understanding Natural Language Instructions from Unseen Maps

Learning to Read Maps: Understanding Natural Language Instructions from Unseen Maps

CIMA: A Large Open Access Dialogue Dataset for Tutoring

Interactive task learning via embodied corrective feedback

Contact Info

Product

Resources

About