Improving LSTM-based Video Description with Linguistic Knowledge Mined from Text

Venugopalan, Subhashini; Hendricks, Lisa Anne; Mooney, Raymond J.; Saenko, Kate

doi:10.48550/arxiv.1604.01729

Cited by 18 publications

(25 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The general framework used also consists of applying one or several CNNs to the input images, followed by a generative LSTM for sentence generation. Several improvements and variations have been proposed to the main model, some examples being the use of additional CNN representations like optical flow (Venugopalan et al, 2015;Yao et al, 2015), Bidirectional LSTMs (BLSTM) in the encoder (Peris et al, 2016), attention mechanisms in the decoder (Yao et al, 2015), hierarchical information (Pan et al, 2016), external linguistic knowledge (Venugopalan et al, 2016) or multi-modal attention mechanisms (Hori et al, 2017), among others.…”

Section: Related Workmentioning

confidence: 99%

“…Similarly to the video description problem, in the egocentric task, we have as input a sequence of frames and we want to output a sentence that describes the input. This problem has already been tackled in the literature on conventional videos with multiple variations (Pan et al, 2016;Venugopalan et al, 2015Venugopalan et al, , 2016Yao et al, 2015;Peris et al, 2016). In the latter work, the frames were encoded by a CNN and a BLSTM network.…”

Section: Egocentric Captioningmentioning

confidence: 99%

See 1 more Smart Citation

Egocentric video description based on temporally-linked sequences

Bolaños

Peris

Casacuberta

et al. 2018

Journal of Visual Communication and Image Representation

View full text Add to dashboard Cite

Egocentric vision consists in acquiring images along the day from a first person point-of-view using wearable cameras. The automatic analysis of this information allows to discover daily patterns for improving the quality of life of the user. A natural topic that arises in egocentric vision is storytelling, that is, how to understand and tell the story relying behind the pictures.In this paper, we tackle storytelling as an egocentric sequences description problem. We propose a novel methodology that exploits information from temporally neighboring events, matching precisely the nature of egocentric sequences. Furthermore, we present a new method for multimodal data fusion consisting on a multi-input attention recurrent network. We also release the EDUB-SegDesc dataset. This is the first dataset for egocentric image sequences description, consisting of 1,339 events with 3,991 descriptions, from 55 days acquired by 11 people. Finally, we prove that our proposal outperforms classical attentional encoder-decoder methods for video description.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Egocentric Captioningmentioning

confidence: 99%

Egocentric video description based on temporally-linked sequences

Bolaños

Peris

Casacuberta

et al. 2018

Journal of Visual Communication and Image Representation

View full text Add to dashboard Cite

show abstract

“…al. [78] proposed an investigation on how linguistic knowledge can aid the video to text generation. In this approach, they amalgamated both neural language model and distributional semantic embedding for the generation of the text in three ways viz.…”

Section: Recentmentioning

confidence: 99%

A Comprehensive Review on Recent Methods and Challenges of Video Description

Singh¹,

Singh²,

Bandyopadhyay³

2020

Preprint

View full text Add to dashboard Cite

Video description involves the generation of the natural language description of actions, events, and objects in the video. There are various applications of video description by filling the gap between languages and vision for visually impaired people, generating automatic title suggestion based on content, browsing of the video based on the content and video-guided machine translation [86] etc.In the past decade, several works had been done in this field in terms of approaches/methods for video description, evaluation metrics, and datasets. For analyzing the progress in the video description task, a comprehensive survey is needed that covers all the phases of video description approaches with a special focus on recent deep learning approaches. In this work, we report a comprehensive survey on the phases of video description approaches, the dataset for video description, evaluation metrics, open competitions for motivating the research on the video description, open challenges in this field, and future research directions. In this survey, we cover the state-of-the-art approaches proposed for each and every dataset with their pros and cons. For the growth of this research domain, the availability of numerous benchmark dataset is a basic need. Further, we categorize all the dataset into two classes: open domain dataset and domain-specific dataset. A brief discussion of the pros and cons of automatic evaluation metrics and human evaluation is also done in this survey. From our survey, we observe that the work in this field is in fast-paced development since the task of video description falls in the intersection of computer vision and natural language processing. But still, the work in the video description is far from saturation stage due to various challenges like the redundancy due to similar frames which affect the quality of visual features, the availability of dataset containing more diverse content and availability of an effective evaluation metric.

show abstract

“…The words in the caption are represented as vectors in a Word Embedding (WE) [34], [35]. The WE is usually learned during the training of the NLVD system [36], or in some cases is a pretrained WE, as in [37]. The decoder is trained to predict the probability of each word in the vocabulary to be the next one in the sentence based on the video encoding and the previous words in the sentence.…”

Section: Related Workmentioning

confidence: 99%

The Role of the Input in Natural Language Video Description

Cascianelli

Devo

Ciarfuglia

et al. 2020

IEEE Trans. Multimedia

View full text Add to dashboard Cite

Natural Language Video Description (NLVD) has recently received strong interest in the Computer Vision, Natural Language Processing (NLP), Multimedia, and Autonomous Robotics communities. The State-of-the-Art (SotA) approaches obtained remarkable results when tested on the benchmark datasets. However, those approaches poorly generalize to new datasets. In addition, none of the existing works focus on the processing of the input to the NLVD systems, which is both visual and textual. In this work, it is presented an extensive study dealing with the role of the visual input, evaluated with respect to the overall NLP performance. This is achieved performing data augmentation of the visual component, applying common transformations to model camera distortions, noise, lighting, and camera positioning, that are typical in real-world operative scenarios. A t-SNE based analysis is proposed to evaluate the effects of the considered transformations on the overall visual data distribution. For this study, it is considered the English subset of Microsoft Research Video Description (MSVD) dataset, which is used commonly for NLVD. It was observed that this dataset contains a relevant amount of syntactic and semantic errors. These errors have been amended manually, and the new version of the dataset (called MSVD-v2) is used in the experimentation. The MSVD-v2 dataset is released to help to gain insight into the NLVD problem.

show abstract

Improving LSTM-based Video Description with Linguistic Knowledge Mined from Text

Cited by 18 publications

References 0 publications

Egocentric video description based on temporally-linked sequences

Egocentric video description based on temporally-linked sequences

A Comprehensive Review on Recent Methods and Challenges of Video Description

The Role of the Input in Natural Language Video Description

Contact Info

Product

Resources

About