Vidiam: Corpus-based Development of a Dialogue Manager for Multimodal Question Answering

Abstract

In this chapter we describe the Vidiam project, which concerns the development of a dialogue management system for multi-modal question answering dialogues as it was carried out in the IMIX project. The approach that was followed is data-driven, that is, corpus-based. Since research in Question Answering Dialog for multi-modal information retrieval is still new, no suitable corpora were available to base a system on. We report on the collection and analysis of three QA dialogue corpora, involving textual followup utterances, multimodal follow-up questions, and speech dialogues. Based on the data, we created a dialogue act typology which helps translate user utterances to practical interactive QA strategies. We then report how we built and evaluated the dialogue manager and its components: dialogue act recognition, interactive QA strategy handling, reference resolution, and multimodal fusion, using off-line analyses of the corpus data.

Follow-up question handling in the IMIX and Ritel systems: A comparative study

“…This paper addresses some of the basic research questions in this area. It continues the line of research laid out by a previous paper (van Schooten and op den Akker, 2005), which discusses the range of basic methods a system can use to handle text-based follow-up utterances (FU), that is, follow-up questions (FQ) and other utterances, as uttered by more or less naive, non-expert users. We define an FQ as any utterance that can in some way be interpreted as a question about the subject domain and is related to previous utterances in the dialogue.…”

“…In previous work (van Schooten and op den Akker, 2005) we distinguished between ‘black-box’ and ‘open’ QA. The difference between IMIX and Ritel is a good example of a black-box QA versus an open QA.…”

“…In this paper, we shall mainly discuss results from four corpora. The first two are the follow-up question corpus (van Schooten and op den Akker, 2005), composed of 575 text FQs, and the multimodal follow-up question corpus (van Schooten and op den Akker, 2007), composed of 202 multimodal (text + gestures) FQs to multimodal (pictures + text) answers. Both are Dutch-language.…”

Chatbots for troubleshooting: A survey

“…Task completion. The metric of task completion measures the percent of DS completion observed at the end of a user-system dialog (Negi et al, 2009;van Schooten & op den Akker, 2011). This metric is particularly appropriate for task-directed dialogs or chats (e.g., booking a flight, buying an item, and filling an online form).…”

“… Dialog precision measures the rate of correct vs. incorrect turns within a conversation Dialog recall measures the rate of correct vs. expected correct turns (as per the systems' knowledge) within a conversation Dialog accuracy measures the rate of correct turns within a conversation Task completion. The metric of task completion measures the percent of DS completion observed at the end of a user‐system dialog (Negi et al, ; van Schooten & op den Akker, ). This metric is particularly appropriate for task‐directed dialogs or chats (e.g., booking a flight, buying an item, and filling an online form).…”