SemEval-2014 Task 6: Supervised Semantic Parsing of Robotic Spatial Commands

Abstract

SemEval-2014 Task 6 aims to advance semantic parsing research by providing a high-quality annotated dataset to compare and evaluate approaches. The task focuses on contextual parsing of robotic commands, in which the additional context of spatial scenes can be used to guide a parser to control a robot arm. Six teams submitted systems using both rule-based and statistical methods. The best performing (hybrid) system scored 92.5% and 90.5% for parsing with and without spatial context. However, the best performing statistical system scored 87.35% and 60.84% respectively, indicating that generalized understanding of commands given to a robot remains challenging, despite the fixed domain used for the task. 'Move the pyramid on the blue cube on the gray one.' Figure 1: Example scene with a contextual spatial command from the Robot Commands Treebank.

SemEval-2017 Task 11: End-User Development using Natural Language

“…Part of this can be explained by the domain-specific nature of previous works (e.g. focus on spatial commands (Dukes, 2014)). …”

“…In 2014, SemEval hosted a task related to the parsing of natural language spatial commands (Dukes, 2014), also targeting a robotics scenario. More specifically, the task proposed the parsing of commands to move a robot arm that moved objects within a spatial region.…”

Natural Language Grounding and Grammar Induction for Robotic Manipulation Commands

“…Therefore, researchers developed unsupervised techniques that learn the semantic categories of words from unlabelled data by exploiting regularities in natural language as in Schütze (1998), Biemann (2009), Socher et al (2012, and Houthooft et al (2016). Similarly, in grammar induction, parsers are commonly trained in a supervised setting on a corpus of annotated grammar trees as presented by Matuszek et al (2013), and Dukes (2014). Other researchers have tackled unsupervised grammar induction from unlabelled sentences as presented by Klein et al (2002) …”

“…), and the grammar structure gets annotated using a tree that connects the different words together (e.g. RCL trees) as presented by Dukes (2014) and Matuszek (2013). The manual annotation of data is a labour intensive task that hinders learning from large corpora, and such labels are not necessarily available for all languages.…”

UW-MRS: Leveraging a Deep Grammar for Robotic Spatial Commands

“…the syntax trees produced by mainstream PCFG parsers and the DRS produced by the Boxer system (Bos, 2008). As a notable exception, Dukes (2014) presents a novel method to produce RCL output directly.…”