DCA: Diversified Co-attention Towards Informative Live Video Commenting

Zhang, Zhihan; Yin, Zhiyi; Ren, Siming; Li, Xinhang; Li, Shicheng

doi:10.1007/978-3-030-60457-8_1

Cited by 10 publications

(8 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We note that reference-based metrics for generation tasks like BLEU and ROUGE are not suitable for evaluation of video comments (Das et al, 2017;Ma et al, 2019;Zhang et al, 2020). Hence we follow (Das et al, 2017) and focus on the ability to rank the correct comment originally appearing at this point in the video over other comments taken from the dataset.…”

Section: Discussionmentioning

confidence: 99%

“…Specifically, we use the code from (Wu et al, 2020), trained on our full dataset with only video frames and surrounding comments as input. The models proposed in (Chaoqun et al, 2020;Zhang et al, 2020) are very recent and their code is not publicly available yet, so we do not consider these as one of our baseline methods. Other older neural architectures such as LSTM are also not included in this study since it is well established that Transformers are the method of choice for modelling multi-modal signals.…”

Section: Methodsmentioning

confidence: 99%

“…They also proposed evaluation metrics and released a publicly accessible training set. This work has served as a benchmark for the most recent approaches (Zhang et al, 2020;Chaoqun et al, 2020;Weiying et al, 2020). In previous work, we reworked the baseline implementationof LiveBot to address several shortcomings in both the original dataset and implementation (Wu et al, 2020).…”

Section: Danmu Generationmentioning

confidence: 99%

See 2 more Smart Citations

Cold Start Problem For Automated Live Video Comments

Wu¹,

Pitié²,

Jones³

2021

Proceedings of the Third Workshop on Multimodal Artificial Intelligence

View full text Add to dashboard Cite

Live video comments, or "danmu", are an emerging feature on Asian online video platforms. Danmu are time-synchronous comments that are overlaid on a video playback. These comments uniquely enrich the experience and engagement of their users, and have become a determining factor in the popularity of videos on these platforms. Similar to the "cold start problem" in recommender systems, a video will only start to attract attention when sufficient danmu comments have been posted on it. We study this video cold start problem and examine how new comments can be generated automatically on less-commented videos. We propose to predict danmu comments to promote user engagement, by exploiting a multi-modal combination of the video visual content, subtitles, audio signals, and any surrounding comments (when they exist). Our method fuses these multiple modalities in a transformer network which is then trained for different comment density scenarios. We evaluate our proposed system through both a retrieval based evaluation method, as well as human judgement. Results show that our proposed system improves significantly over stateof-the-art methods.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: Danmu Generationmentioning

confidence: 99%

See 1 more Smart Citation

Cold Start Problem For Automated Live Video Comments

Wu¹,

Pitié²,

Jones³

2021

Proceedings of the Third Workshop on Multimodal Artificial Intelligence

View full text Add to dashboard Cite

show abstract

“…Previous studies incorporate auxiliary knowledge source like scene graphs or object tags to explicitly indicate the cross-modal mapping. Other researches try to establish fine-grained interaction on cross-modal attention to reinforce the focus from words to their most relevant regions, and vice versa Wang et al, 2019;Messina et al, 2020;Lee et al, 2018;Zhang et al, 2020b;.…”

Section: Self-attention Attention Distributionmentioning

confidence: 99%

Learning Relation Alignment for Calibrated Cross-modal Retrieval

Ren¹,

Lin²,

Zhao³

et al. 2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

Self Cite

View full text Add to dashboard Cite

Despite the achievements of large-scale multimodal pre-training approaches, cross-modal retrieval, e.g., image-text retrieval, remains a challenging task. To bridge the semantic gap between the two modalities, previous studies mainly focus on word-region alignment at the object level, lacking the matching between the linguistic relation among the words and the visual relation among the regions. The neglect of such relation consistency impairs the contextualized representation of image-text pairs and hinders the model performance and the interpretability. In this paper, we first propose a novel metric, Intra-modal Self-attention Distance (ISD), to quantify the relation consistency by measuring the semantic distance between linguistic and visual relations. In response, we present Inter-modal Alignment on Intra-modal Self-attentions (IAIS), a regularized training method to optimize the ISD and calibrate intra-modal self-attentions from the two modalities mutually via inter-modal alignment. The IAIS regularizer boosts the performance of prevailing models on Flickr30k and MS COCO datasets by a considerable margin, which demonstrates the superiority of our approach. 1

show abstract

“…In previous work, we introduced open implementation of Livebot [21] which addresses a number of shortcomings in the original implementation. Follow up work has recently looked at extending the Livebot architecture with different multi-modal fusion approaches [3,27].…”

Section: Related Work 21 Automated Danmu Commentingmentioning

confidence: 99%

Knowing Where and What to Write in Automated Live Video Comments: A Unified Multi-Task Approach

Jones

Pitié

2021

Proceedings of the 2021 International Conference on Multimodal Interaction

View full text Add to dashboard Cite

Live video comments, or "danmu", are an emerging social feature on Asian online video platforms. These time-synchronous comments are overlaid on the video playback and uniquely enrich the viewing experience, engaging hundreds of millions of users in rich community discussions. The presence of danmu comments has become a determining factor for video popularity. Recent work has proposed a model to automatically generate comments, but very little work has so far considered the problem of where to insert the comments in the video timeline. In this work, we propose to address both the what and where of automatic danmu generation, by jointly predicting the danmu comment content to be generated, as well as its optimal insertion point in the video timeline. Our model exploits the video visual content, subtitles, audio signals, and any existing surrounding comments, in one unified architecture and can handle scenarios where the videos are already heavily commented or when the video has no comments yet. Experiments show that our proposed unified framework is in general observed to outperform state-of-the-art comment generation methods. CCS CONCEPTS• Computing methodologies → Natural language generation; Activity recognition and understanding.

show abstract

DCA: Diversified Co-attention Towards Informative Live Video Commenting

Cited by 10 publications

References 23 publications

Cold Start Problem For Automated Live Video Comments

Cold Start Problem For Automated Live Video Comments

Learning Relation Alignment for Calibrated Cross-modal Retrieval

Knowing Where and What to Write in Automated Live Video Comments: A Unified Multi-Task Approach

Contact Info

Product

Resources

About