ROSMI: A Multimodal Corpus for Map-based Instruction-Giving

Katsakioris, Miltiadis Marios; Konstas, Ioannis; Mignotte, Pierre Yves; Hastie, Helen

doi:10.1145/3382507.3418861

Cited by 1 publication

(1 citation statement)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our approach to the destination prediction task is two-fold. The first stage is a data collection for the "Robot Open Street Map Instructions" (ROSMI) (Katsakioris et al, 2020) corpus based on OpenStreetMap (Haklay and Weber, 2008), in which we gather and align NL instructions to their corresponding target destination. We collected 560 NL instruction pairs on 7 maps of different variety and landmarks, in the domain of emergency response using Amazon Mechanical Turk.…”

Section: Introductionmentioning

confidence: 99%

Learning to Read Maps: Understanding Natural Language Instructions from Unseen Maps

Katsakioris¹,

Konstas²,

Mignotte³

et al. 2021

Proceedings of Second International Combined Workshop on Spatial Language Understanding and Grounded Communication for Robotics

Self Cite

View full text Add to dashboard Cite

Robust situated dialog requires the ability to process instructions based on spatial information, which may or may not be available. We propose a model, based on LXMERT, that can extract spatial information from text instructions and attend to landmarks on Open-StreetMap (OSM) referred to in a natural language instruction. Whilst, OSM is a valuable resource, as with any open-sourced data, there is noise and variation in the names referred to on the map, as well as, variation in natural language instructions, hence the need for datadriven methods over rule-based systems. This paper demonstrates that the gold GPS location can be accurately predicted from the natural language instruction and metadata with 72% accuracy for previously seen maps and 64% for unseen maps.

show abstract

Section: Introductionmentioning

confidence: 99%