MultiDM-GCN: Aspect-guided Response Generation in Multi-domain Multi-modal Dialogue System using Graph Convolutional Network

Abstract

In the recent past, dialogue systems have gained immense popularity and have become ubiquitous. During conversations, humans not only rely on languages but seek contextual information through visual contents as well. In every task-oriented dialogue system, the user is guided by the different aspects of a product or service that regulates the conversation towards selecting the product or service. In this work, we present a multi-modal conversational framework for a task-oriented dialogue setup that generates the responses following the different aspects of a product or service to cater to the user's needs. We show that the responses guided by the aspect information provide more interactive and informative responses for better communication between the agent and the user. We first create a Multi-domain Multimodal Dialogue (MDMMD) dataset having conversations involving both text and images belonging to the three different domains, such as restaurants, electronics, and furniture. We implement a Graph Convolutional Network (GCN) based framework that generates appropriate textual responses from the multi-modal inputs. The multi-modal information having both textual and image representation is fed to the decoder and the aspect information for generating aspect guided responses. Quantitative and qualitative analyses show that the proposed methodology outperforms several baselines for the proposed task of aspect-guided response generation.

Data science and AI in FinTech: an overview

“…Other businesses and tasks built on the above interactions include automated identification of interactions and communications and response generation for autonomous, open-domain, context-aware, multi-turn, multi-level, multilingual and knowledge-based dialogue, interactions and communications. Examples are [4,17,36,38]:…”

Dialogue Logic Aware and Key Utterance Decoupling Model for Multi-Party Dialogue Reading Comprehension

“…Currently, M. Firdaus et al [26] and W. Liu et al [27] think that dialogue reading comprehension is mainly used in dialogue systems. W. Wei et al [28] think dialogue reading comprehension is also used for intelligent human-computer interaction systems.…”

Modeling Text-visual Mutual Dependency for Multi-modal Dialog Generation

“…However, these works focus on the QA style visual dialog, rather than the conversation style with which we are more concerned. Another strand of works address multi-turn dialog generation grounded with vision [2,11,18,47,63]. [56] studied the task of image-grounded conversations where utterances are generated about a shared image.…”

“…Different from previous datasets that focus on limited-domain multi-modal conversations such as E-commerce [18,63] or communications grounded in a single image [68], the recently released large open-domain multi-modal dataset OpenViDial [54] contains millions of dialog turns, with each dialog turn associated with its specific visual context (image), in which the dialog utterance takes place, rather than a single image for the whole episode. This better mimics multi-modal dialog generation in the real world.…”