Instruction Clarification Requests in Multimodal Collaborative Dialogue Games: Tasks, and an Analysis of the CoDraw Dataset

Abstract

In visual instruction-following dialogue games, players can engage in repair mechanisms in face of an ambiguous or underspecified instruction that cannot be fully mapped to actions in the world. In this work, we annotate Instruction Clarification Requests (iCRs) in CoDraw, an existing dataset of interactions in a multimodal collaborative dialogue game. We show that it contains lexically and semantically diverse iCRs being produced self-motivatedly by players deciding to clarify in order to solve the task successfully. With 8.8k iCRs found in 9.9k dialogues, CoDraw-iCR (v1) is a large spontaneous iCR corpus, making it a valuable resource for data-driven research on clarification in dialogue. We then formalise and provide baseline models for two tasks: Determining when to make an iCR and how to recognise them, in order to investigate to what extent these tasks are learnable from data. 1 T: above the tree is a cloud with lightning 2 D: small size ? 3 T: it fits right above the tree so the whole cloud is seen and the bolt is just above the top of the tree 4 D: got it and 5 T: to the left of the cloud is a air balloon with a very tip of the top off screen 6 D: is it large or small in size ? 7 T: maybe medium 8 D: done what else 9 T: to the left of the balloon is another regular cloud about one inch from the left 10 D: okay and 11 T: just left of center in the green is a medium girl facing right 12 D: expression of the girl ? what side is she facing ? 13 T: she is standing with a sad face and one hand facing out . she is facing the tree 14 D: got it and