Residual attention-based LSTM for video captioning

Li, Xiangpeng; Zhou, Zhilong; Chen, Lijiang; Gao, Lianli

doi:10.1007/s11280-018-0531-z

Cited by 36 publications

(19 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…X.P. Li et al have developed a novel attention-based framework, called LSTM (Res-ATT [10]). This new model takes advantage of the existing mechanism of attention and integrates residual mapping into a two level LSTM network to avoid the loss of word information previously generated.…”

Section: Video Captioningmentioning

confidence: 99%

“…This new model takes advantage of the existing mechanism of attention and integrates residual mapping into a two level LSTM network to avoid the loss of word information previously generated. The decoder model based on a residual attention has five different parts: a sentence encoder, time attention, visual and sentence fusion layer, residual layer and MLP [10]. The phrase encoder is an LSTM layer exploring important syntactic information from the phrase.…”

Section: Video Captioningmentioning

confidence: 99%

See 1 more Smart Citation

Optimised ARG based Group Activity Recognition for Video Understanding

Kumar

Chauhan

2021

Preprint

View full text Add to dashboard Cite

Video understanding identifies and classifies various actions and events in the video. Many previous works, such as video annotations, have demonstrated promising success in generating general video understanding. However, a fine summary of human activities and their interactions using state-of-the-art video captioning techniques is still difficult to produce. The comprehensive explanation of human actions and collective behaviors is important information for real time CCTV video tracking, medical treatment, sports video analysis etc. This research suggests a form of video understanding that focuses primarily on identifying group activity by learning the similarities between the pair and the actors appearance. In order to measure the similarity between the pair appearances and construct an actor relations map, the Zero Mean Normalized Cross-Correlation (ZNCC) and the zeromean sum of absolute differences(ZSAD) are proposed to allow the graph convolution network (GCN) to learn how to distinguish group actions. We recommend that MNASNet be used as the backbone to retrieve features from any video frame. A visualization model is also developed to visualize every input video frame and predict individual behavior or collective activity with projected bounding boxes on a human object.

show abstract

Section: Video Captioningmentioning

confidence: 99%

Section: Video Captioningmentioning

confidence: 99%

Optimised ARG based Group Activity Recognition for Video Understanding

Kumar

Chauhan

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Attention-based LSTM (Res-ATT) [6]. To describe the video in detail, the mechanism called temporal attention with CNN and LSTM has been applied by them.…”

Section: Et Al Have Developed An Architecture Called Residualmentioning

confidence: 99%

Deep Learning based, a New Model for Video Captioning

Özer¹,

Karapinar²,

Basbug³

et al. 2020

IJACSA

View full text Add to dashboard Cite

Visually impaired individuals face many difficulties in their daily lives. In this study, a video captioning system has been developed for visually impaired individuals to analyze the events through real-time images and express them in meaningful sentences. It is aimed to better understand the problems experienced by visually impaired individuals in their daily lives. For this reason, the opinions and suggestions of the disabled individuals within the Altınokta Blind Association (Turkish organization of blind people) have been collected to produce more realistic solutions to their problems. In this study, MSVD which consists of 1970 YouTube clips has been used as training dataset. First, all clips have been muted so that the sounds of the clips have not been used in the sentence extraction process. The CNN and LSTM architectures have been used to create sentence and experimental results have been compared using BLEU 4, ROUGE-L and CIDEr and METEOR.

show abstract

“…Typically, is used an LSTM-based encoder. This can be a single LSTM [31], a bidirectional LSTM (BiLSTM) [32], or a multilayer LSTM [33]. Using the GRU in the encoder is less common [14].…”

Section: Related Workmentioning

confidence: 99%

“…To improve the performance, attention mechanisms are employed at different points of the encoder-decoder system. In particular, at each word generation step, the decoder takes as input the video features weighted according to their relevance to the next word, based on the previously emitted words [31], [32], [38], [39]. With the same principle, in [40] the attention mechanism is applied to the mean-pooled features from a predefined number of objects tracklets in the video.…”

Section: Related Workmentioning

confidence: 99%

The Role of the Input in Natural Language Video Description

Cascianelli

Devo

Ciarfuglia

et al. 2020

IEEE Trans. Multimedia

View full text Add to dashboard Cite

Natural Language Video Description (NLVD) has recently received strong interest in the Computer Vision, Natural Language Processing (NLP), Multimedia, and Autonomous Robotics communities. The State-of-the-Art (SotA) approaches obtained remarkable results when tested on the benchmark datasets. However, those approaches poorly generalize to new datasets. In addition, none of the existing works focus on the processing of the input to the NLVD systems, which is both visual and textual. In this work, it is presented an extensive study dealing with the role of the visual input, evaluated with respect to the overall NLP performance. This is achieved performing data augmentation of the visual component, applying common transformations to model camera distortions, noise, lighting, and camera positioning, that are typical in real-world operative scenarios. A t-SNE based analysis is proposed to evaluate the effects of the considered transformations on the overall visual data distribution. For this study, it is considered the English subset of Microsoft Research Video Description (MSVD) dataset, which is used commonly for NLVD. It was observed that this dataset contains a relevant amount of syntactic and semantic errors. These errors have been amended manually, and the new version of the dataset (called MSVD-v2) is used in the experimentation. The MSVD-v2 dataset is released to help to gain insight into the NLVD problem.

show abstract

Residual attention-based LSTM for video captioning

Cited by 36 publications

References 33 publications

Optimised ARG based Group Activity Recognition for Video Understanding

Optimised ARG based Group Activity Recognition for Video Understanding

Deep Learning based, a New Model for Video Captioning

The Role of the Input in Natural Language Video Description

Contact Info

Product

Resources

About