Learning to Discretely Compose Reasoning Module Networks for Video Captioning

Tan, Ganchao; Liu, Daqing; Wang, Meng; Zha, Zheng-Jun

doi:10.48550/arxiv.2007.09049

Cited by 4 publications

(4 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The model of RMN is from the literature [15]. In the ablation experiments in Table 1, the effectiveness of the proposed gated fusion mechanism and sentence length modulation loss function method is proved, where "RMN" corresponds to the experimental results of the literature [15], The methods proposed in this paper are based on the benchmarking paper model. Firstly, the effectiveness of the gated fusion mechanism is verified.…”

Section: Analysis Of Ablation Experiments Resultsmentioning

confidence: 99%

Video Description Method based on Semantic Information Filtering and Sentence Length Modulation

Wang¹,

Cai²,

Zhou³

et al. 2023

Atlantis Highlights in Computer Sciences

View full text Add to dashboard Cite

In the current video description task, the spatial redundancy information in the video features is usually not effectively eliminated, and the commonly used loss function is composed of the logarithm of the probability of the correct word of the target, and the long sentences formed often bring great losses to the model. If the sentence length generated by the optimization of the log-likelihood loss function is too short, the description semantics will be incomplete and the accuracy will not be high. This paper proposes a video description method based on semantic information filtering and sentence length modulation to solve the above problems. Firstly, the model introduces a gated fusion mechanism, which removes redundant information in the semantic information of video features by screening the semantic features of the video, reduces the interference of redundant information on the generated description, and improves the accuracy of the description. Secondly, a new sentence length modulation loss function is proposed, which modulates the cross-entropy loss function with the label sentence length, which alleviates the tendency of the model to generate short sentences, and makes the semantics of the generated description close to the label, thereby improving the accuracy of the description. The experimental results on the MSVD dataset, which is widely used in this field, show that the method in this paper can significantly improve the accuracy of generating video descriptions, and all indicators are significantly better than existing models.

show abstract

Section: Analysis Of Ablation Experiments Resultsmentioning

confidence: 99%

Video Description Method based on Semantic Information Filtering and Sentence Length Modulation

Wang¹,

Cai²,

Zhou³

et al. 2023

Atlantis Highlights in Computer Sciences

View full text Add to dashboard Cite

show abstract

“…We use RMN [59] as our captioning module, and extract features in the same manner as mentioned by the authors. Note that the localization module is kept frozen throughout the training process.…”

Section: Video Captioningmentioning

confidence: 99%

Tragedy Plus Time: Capturing Unintended Human Activities from Weakly-labeled Videos

Chakravarthy¹,

Fang²,

Yang³

2022

Preprint

View full text Add to dashboard Cite

In videos that contain actions performed unintentionally, agents do not achieve their desired goals. In such videos, it is challenging for computer vision systems to understand high-level concepts such as goal-directed behavior, an ability present in humans from a very early age. Inculcating this ability in artificially intelligent agents would make them better social learners by allowing them to evaluate human action under a teleological lens. To validate this ability of deep learning models to perform this task, we curate the W-Oops dataset, built upon the Oops dataset [15]. W-Oops consists of 2,100 unintentional human action videos, with 44 goal-directed and 30 unintentional video-level activity labels collected through human annotations. Due to the expensive segment annotation procedure, we propose a weakly supervised algorithm for localizing the goaldirected as well as unintentional temporal regions in the video leveraging solely video-level labels. In particular, we employ an attention mechanism based strategy that predicts the temporal regions which contributes the most to a classification task. Meanwhile, our designed overlap regularization allows the model to focus on distinct portions of the video for inferring the goal-directed and unintentional activity, while guaranteeing their temporal ordering. Extensive quantitative experiments verify the validity of our localization method. We further conduct a video captioning experiment which demonstrates that the proposed localization module does indeed assist teleological action understanding. Project website can be found at: https://asuapg.github.io/TragedyPlusTime.

show abstract

“…Along this line, many works improve video captioning by either designing a better visual encoder [16,40,12,1,15] or language decoder [79,50,73]. Of particular interest are [28,63] which combine a sequence-to-sequence model with old-fashioned template methods for model grounding video caption generation.…”

Section: Related Workmentioning

confidence: 99%

Neuro-Symbolic Representations for Video Captioning: A Case for Leveraging Inductive Biases for Vision and Language

Akbari,

Palangi,

Yang

et al. 2020

Preprint

View full text Add to dashboard Cite

Neuro-symbolic representations have proved effective in learning structure information in vision and language. In this paper, we propose a new model architecture for learning multi-modal neuro-symbolic representations for video captioning. Our approach uses a dictionary learningbased method of learning relations between videos and their paired text descriptions. We refer to these relations as relative roles and leverage them to make each token role-aware using attention. This results in a more structured and interpretable architecture that incorporates modality-specific inductive biases for the captioning task. Intuitively, the model is able to learn spatial, temporal, and cross-modal relations in a given pair of video and text. The disentanglement achieved by our proposal gives the model more capacity to capture multi-modal structures which result in captions with higher quality for videos. Our experiments on two established video captioning datasets verifies the effectiveness of the proposed approach based on automatic metrics. We further conduct a human evaluation to measure the grounding and relevance of the generated captions and observe consistent improvement for the proposed model. The codes and trained models can be found at https://github.com/hassanhub/R3Transformer.

show abstract

Learning to Discretely Compose Reasoning Module Networks for Video Captioning

Cited by 4 publications

References 27 publications

Video Description Method based on Semantic Information Filtering and Sentence Length Modulation

Video Description Method based on Semantic Information Filtering and Sentence Length Modulation

Tragedy Plus Time: Capturing Unintended Human Activities from Weakly-labeled Videos

Neuro-Symbolic Representations for Video Captioning: A Case for Leveraging Inductive Biases for Vision and Language

Contact Info

Product

Resources

About