Image Captioning With End-to-End Attribute Detection and Subsequent Attributes Prediction

Huang, Yiqing; Chen, Jiansheng; Ouyang, Wanli; Wan, Wei; Xue, Youze

doi:10.1109/tip.2020.2969330

Cited by 78 publications

(35 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The authors also have proposed to use the Convolutional GRU to calculate the assignment to investigate the spatio-temporal correlation at each time-stamp between the successive frames. Huang et al [44] have proposed an image captioning model which comes with End-to-End Attribute Detection and Subsequent Attributes Prediction. In this model, features are extracted using ResNet-101 based Faster-RCNN.…”

Section: Feature Extractionmentioning

confidence: 99%

See 1 more Smart Citation

Exploring Video Captioning Techniques: A Comprehensive Survey on Deep Learning Methods

et al. 2021

View full text Add to dashboard Cite

Video captioning is an automated collection of natural language phrases that explains the contents in video frames. Because of the incomparable performance of deep learning in the field of computer vision and natural language processing in recent years, research in this field has been exponentially increased throughout past decades. Numerous approaches, datasets, and measurement metrics have been introduced in the literature, calling for a systematic survey to guide research efforts in this exciting new direction. Through the statistical analysis, this survey paper focuses mostly on state-of-the-art approaches, emphasizing deep learning models, assessing benchmark datasets in several parameters, and classifying the pros and cons of the various evaluation metrics based on the previous works in the deep learning field. This survey shows the most used variants of neural networks for visual and spatio-temporal feature extraction as well as language generation model. The results show that ResNet and VGG as visual feature extractor and 3D convolutional neural network as spatio-temporal feature extractor are mostly used. Besides that, Long Short Term Memory (LSTM) has been mainly used as the language model. However, nowadays, the Gated Recurrent Unit (GRU) and Transformer are slowly replacing LSTM. Regarding dataset usage, so far, MSVD and MSR-VTT are very much dominant due to be part of outstanding results among various captioning models. From 2015 to 2020, with all major datasets, some models such as, Inception-Resnet-v2 + C3D + LSTM, ResNet-101 + I3D + Transformer, ResNet-152 + ResNext-101 (R3D) + (LSTM, GAN) have achieved by far best results in video captioning. Despite rapid advancement, our survey reveals that video captioning research-work still has a lot to develop in accessing the full potential of deep learning for classifying and captioning a large number of activities, as well as creating large datasets covering diversified training video samples.

show abstract

Section: Feature Extractionmentioning

confidence: 99%

“…Wang et al [126] have proposed model has two layers, and each layer has one GRU in the decoding phase. Huang et al [44] have proposed to implement Two layers SAP-LSTM for decoding purposes.…”

Section: Sentence Generationmentioning

confidence: 99%

Exploring Video Captioning Techniques: A Comprehensive Survey on Deep Learning Methods

et al. 2021

View full text Add to dashboard Cite

show abstract

“…These works could be sorted based on the attention network structure and the attention weight calculation method. (1) Single layer vs. multi-layer (see, for example, studies [3,3,4,[19][20][21]) employs a single layer implement of an attention mechanism by taking the hidden state as the query vector to extract visual features at each step, while the studies [8,16,17,22,23] chose a multi-layer attention implementation in their decoder. (2) Involving extra clues in attention weights or not-for example, studies [11,24,25] obtained their attention weights only from previous attention calculations, while the authors of [18] combined geometry clues with previously calculated weights into final attention weights.…”

Section: Related Workmentioning

confidence: 99%

Learn and Tell: Learning Priors for Image Caption Generation

2020

View full text Add to dashboard Cite

In this work, we propose a novel priors-based attention neural network (PANN) for image captioning, which aims at incorporating two kinds of priors, i.e., the probabilities being mentioned for local region proposals (PBM priors) and part-of-speech clues for caption words (POS priors), into a visual information extraction process at each word prediction. This work was inspired by the intuitions that region proposals have different inherent probabilities for image captioning, and that the POS clues bridge the word class (part-of-speech tag) with the categories of visual features. We propose new methods to extract these two priors, in which the PBM priors are obtained by computing the similarities between the caption feature vector and local feature vectors, while the POS priors are predicated at each step of word generation by taking the hidden state of the decoder as input. After that, these two kinds of priors are further incorporated into the PANN module of the decoder to help the decoder extract more accurate visual information for the current word generation. In our experiments, we qualitatively analyzed the proposed approach and quantitatively evaluated several captioning schemes with our PANN on the MS-COCO dataset. Experimental results demonstrate that our proposed method could achieve better performance as well as the effectiveness of the proposed network for image captioning.

show abstract

“…The application of image captioning is very broad and significant, especially in the field of human and computer interaction. Apart from this, practical image captioning helps disabled people interact with normal people [6]. It is also used to create multimedia content descriptions, assist e-commerce companies with digital marketing, and create online news stories for news content narratives [7].…”

Section: Introductionmentioning

confidence: 99%

A Rapid Review of Image Captioning

Adriyendi

2021

JITeCS

View full text Add to dashboard Cite

Image captioning is an automatic process for generating text based on the content observed in an image. We do review, create framework, and build application model. We review image captioning into 4 categories based on input model, process model, output model, and lingual image caption. Input model is based on criteria caption, method, and dataset. Process model is based on type of learning, encoder-decoder, image extractor, and metric evaluation. Output model based on architecture, features extraction, feature aping, model, and number of caption. Lingual image caption based on language model with 2 groups: bilingual image caption and cross-language image caption. We also design framework with 3 framework model. Furthermore, we also build application with 3 application models. We also provide research opinions on trends and future research that can be developed with image caption generation. Image captioning can be further developed on computer vision versusÂ human vision.

show abstract

Image Captioning With End-to-End Attribute Detection and Subsequent Attributes Prediction

Cited by 78 publications

References 31 publications

Exploring Video Captioning Techniques: A Comprehensive Survey on Deep Learning Methods

Exploring Video Captioning Techniques: A Comprehensive Survey on Deep Learning Methods

Learn and Tell: Learning Priors for Image Caption Generation

A Rapid Review of Image Captioning

Contact Info

Product

Resources

About