From Show to Tell: A Survey on Deep Learning-Based Image Captioning

Stefanini, Matteo; Cornia, Marcella; Baraldi, Lorenzo; Cascianelli, Silvia; Fiameni, Giuseppe; Cucchiara, Rita

doi:10.1109/tpami.2022.3148210

Cited by 178 publications

(70 citation statements)

References 184 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Although the solution to the problem of automated image description generation yields acceptable results, there is still space for improvement because certain photos are not well described. Metric-oriented focal mechanism (MFM) [23] e survey presented by [24] provides an in-depth analysis of picture captioning methods, including image encoding and text generation, as well as training methodologies, datasets, and assessment measures. e study concludes that there are still many unresolved issues including accuracy, robustness, and generalization outcomes.…”

Section: Image Captioning Methodologiesmentioning

confidence: 99%

Caption Generation Based on Emotions Using CSPDenseNet and BiLSTM with Self-Attention

S¹,

K²,

Kaliappan

et al. 2022

Applied Computational Intelligence and Soft Computing

View full text Add to dashboard Cite

Automatic image caption generation is an intricate task of describing an image in natural language by gaining insights present in an image. Featuring facial expressions in the conventional image captioning system brings out new prospects to generate pertinent descriptions, revealing the emotional aspects of the image. e proposed work encapsulates the facial emotional features to produce more expressive captions similar to human-annotated ones with the help of Cross Stage Partial Dense Network (CSPDenseNet) and Self-attentive Bidirectional Long Short-Term Memory (BiLSTM) network. e encoding unit captures the facial expressions and dense image features using a Facial Expression Recognition (FER) model and CSPDense neural network, respectively. Further, the word embedding vectors of the ground truth image captions are created and learned using the Word2Vec embedding technique. en, the extracted image feature vectors and word vectors are fused to form an encoding vector representing the rich image content. e decoding unit employs a self-attention mechanism encompassed with BiLSTM to create more descriptive and relevant captions in natural language. e Flickr11k dataset, a subset of the Flickr30k dataset is used to train, test, and evaluate the present model based on ve benchmark image captioning metrics. ey are BiLingual Evaluation Understudy (BLEU), Metric for Evaluation of Translation with Explicit Ordering (METEOR), Recall-Oriented Understudy for Gisting Evaluation (ROGUE), Consensus-based Image Description Evaluation (CIDEr), and Semantic Propositional Image Caption Evaluation (SPICE). e experimental analysis indicates that the proposed model enhances the quality of captions with 0.6012(BLEU-1), 0.3992(BLEU-2), 0.2703(BLEU-3), 0.1921(BLEU-4), 0.1932(METEOR), 0.2617(CIDEr), 0.4793(ROUGE-L), and 0.1260(SPICE) scores, respectively, using additive emotional characteristics and behavioral components of the objects present in the image.

show abstract

Section: Image Captioning Methodologiesmentioning

confidence: 99%

Caption Generation Based on Emotions Using CSPDenseNet and BiLSTM with Self-Attention

S¹,

K²,

Kaliappan

et al. 2022

Applied Computational Intelligence and Soft Computing

View full text Add to dashboard Cite

show abstract

“…Image captioning requires both a deep understanding of the visual content within an image, its objects, attributes, and relationships and the capability of a language model to generate syntactically and semantically correct descriptions. In particular, the language model is asked to generate a sentence conditioned on the image representation, whose role is key for obtaining satisfactory results [41].…”

Section: Related Workmentioning

confidence: 99%

The Unreasonable Effectiveness of CLIP Features for Image Captioning: An Experimental Analysis

Barraco

Cornia

Cascianelli

et al. 2022

2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)

Self Cite

View full text Add to dashboard Cite

Generating textual descriptions from visual inputs is a fundamental step towards machine intelligence, as it entails modeling the connections between the visual and textual modalities. For years, image captioning models have relied on pre-trained visual encoders and object detectors, trained on relatively small sets of data. Recently, it has been observed that large-scale multi-modal approaches like CLIP (Contrastive Language-Image Pre-training), trained on a massive amount of image-caption pairs, provide a strong zero-shot capability on various vision tasks. In this paper, we study the advantage brought by CLIP in image captioning, employing it as a visual encoder. Through extensive experiments, we show how CLIP can significantly outperform widely-used visual encoders and quantify its role under different architectures, variants, and evaluation protocols, ranging from classical captioning performance to zero-shot transfer.

show abstract

“…Online evaluation. Finally, we also report the performance of our method on the online COCO test server 1 . In this case, we also employ an ensemble of four models trained with the mesh-like connectivity.…”

Section: Comparison With the State Of The Artmentioning

confidence: 99%

“…Image captioning has received a relevant interest in the last few years, as describing images in natural language is a fundamental step to model the interconnections between the visual and textual modalities [1]. Early works on this research line were based on the usage of convolutional neural networks [2], [3], [4], [5] -usually pre-trained to solve classification tasks -or object detectors [6], [7], [8] as visual feature extractors, and recurrent neural networks as autoregressive language models [2], [3], [6], [9], [10].…”

Section: Introductionmentioning

confidence: 99%

CaMEL: Mean Teacher Learning for Image Captioning

Barraco¹,

Stefanini²,

Cornia³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Describing images in natural language is a fundamental step towards the automatic modeling of connections between the visual and textual modalities. In this paper we present CaMEL, a novel Transformer-based architecture for image captioning. Our proposed approach leverages the interaction of two interconnected language models that learn from each other during the training phase. The interplay between the two language models follows a mean teacher learning paradigm with knowledge distillation. Experimentally, we assess the effectiveness of the proposed solution on the COCO dataset and in conjunction with different visual feature extractors. When comparing with existing proposals, we demonstrate that our model provides stateof-the-art caption quality with a significantly reduced number of parameters. According to the CIDEr metric, we obtain a new state of the art on COCO when training without using external data. The source code and trained models are publicly available at: https://github.com/aimagelab/camel.

show abstract

From Show to Tell: A Survey on Deep Learning-Based Image Captioning

Cited by 178 publications

References 184 publications

Caption Generation Based on Emotions Using CSPDenseNet and BiLSTM with Self-Attention

Caption Generation Based on Emotions Using CSPDenseNet and BiLSTM with Self-Attention

The Unreasonable Effectiveness of CLIP Features for Image Captioning: An Experimental Analysis

CaMEL: Mean Teacher Learning for Image Captioning

Contact Info

Product

Resources

About