Semi-Supervised Learning for Video Captioning

Abstract

Deep neural networks have made great success on video captioning in supervised learning setting. However, annotating videos with descriptions is very expensive and time-consuming. If the video captioning algorithm can benefit from a large number of unlabeled videos, the cost of annotation can be reduced. In the proposed study, we make the first attempt to train the video captioning model on labeled data and unlabeled data jointly, in a semisupervised learning manner. For labeled data, we train them with the traditional crossentropy loss. For unlabeled data, we leverage a self-critical policy gradient method with the difference between the scores obtained by Monte-Carlo sampling and greedy decoding as the reward function, while the scores are the K-L divergence between output distributions of original video data and augmented video data. The final loss is the weighted sum of losses obtained by labeled data and unlabeled data. Experiments conducted on VATEX, MSR-VTT and MSVD dataset demonstrate that the introduction of unlabeled data can improve the performance of the video captioning model. The proposed semi-supervised learning algorithm also outperforms several state-of-the-art semisupervised learning approaches.

Leveraging Weighted Cross-Graph Attention for Visual and Semantic Enhanced Video Captioning Network

“…However, these models ignore the saliency information present in a video since the visual regions are often ambiguous and over-sampled, which makes the captioning task more challenging. Even some works (Lin, Gan, and Wang 2020;Liu et al 2019) have focused on the ROI and visual grounding by decomposing expressions into logic structures. Yet, these methods are not able to preserve the semantic structure of salient regions to generate human-like descriptions.…”