Multimodal Dialog System: Relational Graph-based Context-aware Question Understanding

Abstract

Multimodal dialog system has attracted increasing attention from both academia and industry over recent years. Although existing methods have achieved some progress, they are still confronted with challenges in the aspect of question understanding (i.e., user intention comprehension). In this paper, we present a relational graph-based context-aware question understanding scheme, which enhances the user intention comprehension from local to global. Specifically, we first utilize multiple attribute matrices as the guidance information to fully exploit the product-related keywords from each textual sentence, strengthening the local representation of user intentions. Afterwards, we design a sparse graph attention network to adaptively aggregate effective context information for each utterance, completely understanding the user intentions from a global perspective. Moreover, extensive experiments over a benchmark dataset show the superiority of our model compared with several state-of-the-art baselines. CCS CONCEPTS• Computing methodologies → Discourse, dialogue and pragmatics.

Multimodal Dialog Systems with Dual Knowledge-enhanced Generative Pretrained Language Model

“…Moreover, they released a large-scale multimodal dialog dataset in the context of online fashion shopping, named MMD, which significantly promotes the research progress on multimodal dialog systems. In particular, several efforts further explore the semantic relation in the multimodal dialog context and incorporate knowledge based on the framework of MHRED [2], [3], [4], [5], [6], [7]. For example, Liao et al [5] developed a taxonomy-based visual semantic learning module to capture the fine-grained semantics (e.g., the category and attributes of a product) in product images, and introduced a memory network to integrate the knowledge of fashion style tips.…”

“…where Z enc l ∈ R M ×D refers to the output of l-th encoder layer, and Z enc 0 is obtained by the aforementioned position-wise embedding layer in Eqn. (2). Z S l ∈ R M ×D is the intermediate output of MSA in the l-th encoder layer.…”

“…If it appears, we select its attributes A p as the related knowledge. Notably, we here only consider attributes rather than images due to the fact that the attribute knowledge is essential to understanding user's intentions and generating the text response [2]. In this vein, we can obtain the overall knowledge set involved with the textual context, denoted as…”

“…In particular, we first obtain the initial embedding of X t , denoted as E t ∈ R Nt×D , by the position-wise embedding layer of BART in Eqns. ( 1) and (2). Thereafter, to capture the semantic representation, we feed the initial embedding E t into the bidirectional encoder B e of BART defined in Eqn.…”

“…Existing multimodal task-oriented dialog systems mainly adopt the encoder-decoder framework for text response generation. In particular, recent studies have recognized the pivotal role of the knowledge base for multimodal dialog systems, and designed various schemes for incorporating knowledge to enhance the user's intention understanding [2], [3], [4], [5], [6], [7], [8], [9]. Although they have achieved significant progress, these research efforts suffer from two key limitations.…”