End-to-end Audio Visual Scene-aware Dialog Using Multimodal Attention-based Video Features

Abstract

Dialog systems need to understand dynamic visual scenes in order to have conversations with users about the objects and events around them. Scene-aware dialog systems for real-world applications could be developed by integrating state-ofthe-art technologies from multiple research areas, including: end-to-end dialog technologies, which generate system responses using models trained from dialog data; visual question answering (VQA) technologies, which answer questions about images using learned image features; and video description technologies, in which descriptions/captions are generated from videos using multimodal information. We introduce a new dataset of dialogs about videos of human behaviors. Each dialog is a typed conversation that consists of a sequence of 10 question-and-answer (QA) pairs between two Amazon Mechanical Turk (AMT) workers. In total, we collected dialogs on ∼ 9, 000 videos. Using this new dataset, we trained an end-toend conversation model that generates responses in a dialog about a video. Our experiments demonstrate that using multimodal features that were developed for multimodal attention-based video description enhances the quality of generated dialog about dynamic scenes (videos). Our dataset, model code and pretrained models will be publicly available for a new Video Scene-Aware Dialog challenge.

Factor Graph Attention

“…For VQA v1.0 we increase validation set accuracy from 57.0 to 57.3 (no tuning) by replacing the alternating and parallel attention [33]. For AVSD, we improve Hori et al [19] which report a CIDEr score of 0.733 to 0.806. We used FGA to attend to all video cues as well as the question.…”

A Simple Baseline for Audio-Visual Scene-Aware Dialog

“…We evaluate the proposed approach on the recently introduced and challenging audio-visual sceneaware dataset, and demonstrate the key features that permit to outperform the current state-of-the-art by more than 20% on CIDEr. Recent work on audio-visual scene aware dialog [2,25] partly addresses this shortcoming and proposes a novel Question: what color is the rag ?Answer: it appears to be white .…”

Learning Question-Guided Video Representation for Multi-Turn Video Question Answering

“…This suggests the effectiveness of our proposed question-guided video representations for VideoQA. When comparing Table 2: Comparison with existing approaches: Naïve Fusion (Alamri et al, 2019b;Zhuang et al, 2019), Attentional Fusion (Hori et al, 2018;Zhuang et al, 2019), Multi-Source Sequence-to-Sequence model (Pasunuru and Bansal, 2019), Modified Attentional Fusion with Maximum Mutual Information objective (Zhuang et al, 2019) and Hierarchical Attention with pre-trained embedding (Le et al, 2019), on the AVSD public test set. For each approach, we report its corpus-wide scores on BLEU-1 through BLEU-4, METEOR, ROUGE-L and CIDEr.…”