Bridging Text and Video: A Universal Multimodal Transformer for Video-Audio Scene-Aware Dialog

Li, Zekang; Li, Zongjia; Zhang, Jinchao; Feng, Yang; Niu, Cheng; Zhou, Jie

doi:10.48550/arxiv.2002.00163

Cited by 9 publications

(8 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition to our full model, we also show results without the object level features referred to as 'CoMVT (Scene feats only)' in Table 1, as this is more similar to previous multimodal models [25,42,48,77], as well as show the effects without BERT pretraining for the text stream and the MLM loss. For each model including the baselines, we perform grid search on learning rates and report the test performance of the best models in the validation set.…”

Section: Baselinessupporting

confidence: 69%

“…Related to our work is the task of scene-aware dialog prediction [2,33], where the goal is to answers questions grounded to a video clip input, given a manually created dialog history. A number of works show promising results on this task [14,32,42,45,48]. In contrast, we predict future utterances from videos using the natural speech present in the video itself.…”

Section: Related Workmentioning

confidence: 99%

“…A large number of multimodal architectures focus on late fusion of modalities, with popular choices being summation, concatenation and canonical correlation analysis [9,74,78]. More recent works [25,48] explore deeper interactions between video frame features and features from other modalities, by fusing modalities earlier, at the input level itself. In these works, however, inputs from multiple modalities are fed into a single transformer, with 'modality specific' encodings to distinguish between the modalities.…”

Section: Related Workmentioning

confidence: 99%

“…Text only Baseline: For the text only baseline, we use BERT [19], which is the winning model in the Eighth Dialog System Technology Challenge for response prediction in text only dialog benchmarks [29]. Single-Stream Multimodal Baseline: We also implement a single stream transformer operating on a single multimodal input stream, which is the most widely used framework for video encoding with multimodal inputs [25,42,48,77]. We adopt the architecture used in [48] and train the network using the same next utterance prediction loss for FUP.…”

Section: Baselinesmentioning

confidence: 99%

“…Single-Stream Multimodal Baseline: We also implement a single stream transformer operating on a single multimodal input stream, which is the most widely used framework for video encoding with multimodal inputs [25,42,48,77]. We adopt the architecture used in [48] and train the network using the same next utterance prediction loss for FUP. Note that this architecture is slightly different from that of BERT, and hence we cannot use pre-trained BERT weights.…”

Section: Baselinesmentioning

confidence: 99%

See 4 more Smart Citations

Look Before you Speak: Visually Contextualized Utterances

Seo¹,

Nagrani²,

Schmid³

2020

Preprint

View full text Add to dashboard Cite

Think of a really tasty thing. It will likely be flat again. It makes it feel healthier. Now slip that nut back on and screw it down. It's going to take about five minutes. … Transcript: I'm going to go ahead and slip that into place and I'm going to make note of which way the arrow is going in relation to the arrow on our guard. They both need to be going the same direction next. Prediction Next utterance candidates Input Video ✔ Figure 1: Visually Contextualised Future Utterance Prediction. Given an instructional video with paired text and video data, we predict the next utterance in the video using a Co-attentional Multimodal Video Transformer. Our model trained on this task also achieves state-of-the-art performance on downstream VideoQA benchmarks.

show abstract

Section: Baselinessupporting

confidence: 69%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Baselinesmentioning

confidence: 99%

Section: Baselinesmentioning

confidence: 99%

See 3 more Smart Citations

Look Before you Speak: Visually Contextualized Utterances

Seo¹,

Nagrani²,

Schmid³

2020

Preprint

View full text Add to dashboard Cite

show abstract

VGNMN: Video-grounded Neural Module Networks for Video-Grounded Dialogue Systems

Lê¹,

Chen²,

Hoi³

2022

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

Neural module networks (NMN) have achieved success in image-grounded tasks such as Visual Question Answering (VQA) on synthetic images. However, very limited work on NMN has been studied in the video-grounded dialogue tasks. These tasks extend the complexity of traditional visual tasks with the additional visual temporal variance and language cross-turn dependencies. Motivated by recent NMN approaches on image-grounded tasks, we introduce Videogrounded Neural Module Network (VGNMN) to model the information retrieval process in video-grounded language tasks as a pipeline of neural modules. VGNMN first decomposes all language components in dialogues to explicitly resolve any entity references and detect corresponding action-based inputs from the question. The detected entities and actions are used as parameters to instantiate neural module networks and extract visual cues from the video. Our experiments show that VGNMN can achieve promising performance on a challenging video-grounded dialogue benchmark as well as a video QA benchmark.

show abstract

DVD: A Diagnostic Dataset for Multi-step Reasoning in Video Grounded Dialogue

Lê

Sankar

Moon

et al. 2021

Preprint

View full text Add to dashboard Cite

A video-grounded dialogue system is required to understand both dialogue, which contains semantic dependencies from turn to turn, and video, which contains visual cues of spatial and temporal scene variations. Building such dialogue systems is a challenging problem involving complex multimodal and temporal inputs, and studying them independently is hard with existing datasets. Existing benchmarks do not have enough annotations to help analyze dialogue systems and understand their linguistic and visual reasoning capability and limitations in isolation. These benchmarks are also not explicitly designed to minimise biases that models can exploit without actual reasoning. To address these limitations, in this paper, we present a diagnostic dataset that can test a range of reasoning abilities on videos and dialogues. The dataset is designed to contain minimal biases and has detailed annotations for the different types of reasoning each question requires, including cross-turn video interval tracking and dialogue object tracking. We use our dataset to analyze several dialogue system approaches, providing interesting insights into their abilities and limitations. In total, the dataset contains 10 instances of 10-round dialogues for each of ∼ 11k synthetic videos, resulting in more than 100k dialogues and 1M question-answer pairs. Our code and dataset will be made public.

show abstract

Bridging Text and Video: A Universal Multimodal Transformer for Video-Audio Scene-Aware Dialog

Cited by 9 publications

References 14 publications

Look Before you Speak: Visually Contextualized Utterances

Look Before you Speak: Visually Contextualized Utterances

VGNMN: Video-grounded Neural Module Networks for Video-Grounded Dialogue Systems

DVD: A Diagnostic Dataset for Multi-step Reasoning in Video Grounded Dialogue

Contact Info

Product

Resources

About