Dual Graph Convolutional Networks with Transformer and Curriculum Learning for Image Captioning

Dong, Xinzhi; Long, Chengjiang; Xu, Wenju; Xiao, Chunxia

doi:10.48550/arxiv.2108.02366

Cited by 6 publications

(5 citation statements)

References 50 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Fan et al [51] consistently demonstrated strong performance across various metrics, making it a notable candidate for effective image captioning. Dong et al [53] and Yao et al [57] also performed consistently well across multiple metrics.…”

Section: Flicker and Coco Image Captioningmentioning

confidence: 73%

An Extensive Analysis of Image Captioning Models, Evaluation Measures, and Datasets

Jaiswal,

Pallthadka,

Chinchewadi

et al. 2023

IJCERT

View full text Add to dashboard Cite

The difficult interdisciplinary endeavour of creating insightful and detailed captions for images is known as "image captioning," and it lies at the nexus of computer vision and natural language processing. We give a comprehensive examination of datasets, evaluation metrics, and image captioning models in this paper. We present a thorough review of popular image captioning models, from conventional methods to the most recent developments utilizing deep learning and attention mechanisms. We examine the design, underlying assumptions, and capabilities of these models, emphasizing how they help produce logical and contextually appropriate captions. Furthermore, we analyses in detail well-known evaluation metrics as BLEU, METEOR, ROUGE, and CIDEr, clarifying their importance in evaluating generated caption quality against ground truth references. Additionally, we talk about the critical role that datasets play in image captioning research, with particular attention to prominent datasets like as COCO, Flickr30k, and Conceptual Captions. We investigate these dataset's diversity, volume, and annotations, emphasizing their impact on model evaluation and training. Our objective is to furnish scholars, professionals, and amateurs with an invaluable tool for comprehending the state of image captioning, so facilitating the creation of inventive models and enhanced assessment techniques.

show abstract

Section: Flicker and Coco Image Captioningmentioning

confidence: 73%

An Extensive Analysis of Image Captioning Models, Evaluation Measures, and Datasets

Jaiswal,

Pallthadka,

Chinchewadi

et al. 2023

IJCERT

View full text Add to dashboard Cite

show abstract

“…Encoder-decoder based approach is a most widely used for machine translation and image caption generation which is based upon deep neural networks. A dual graph convolution network based is proposed in [33] and NIC (Neural Image Caption) model based on encoder-decoder architecture is in [27]. This one is a simple model where CNN is used as a encoder, and in the decoder end LSTM and RNN are used for image caption generation.…”

Section: Literature Surveymentioning

confidence: 99%

An Efficient Deep Learning based Hybrid Model for Image Caption Generation

Kaur¹,

Kaur²

2023

IJACSA

View full text Add to dashboard Cite

In the recent yeas, with the increase in the use of different social media platforms, image captioning approach play a major role in automatically describe the whole image into natural language sentence. Image captioning plays a significant role in computer-based society. Image captioning is the process of automatically generating the natural language textual description of the image using artificial intelligence techniques. Computer vision and natural language processing are the key aspect of the image processing system. Convolutional Neural Network (CNN) is a part of computer vision and used object detection and feature extraction and on the other side Natural Language Processing (NLP) techniques help in generating the textual caption of the image. Generating suitable image description by machine is challenging task as it is based upon object detection, location and their semantic relationships in a human understandable language such as English. In this paper our aim to develop an encoder-decoder based hybrid image captioning approach using VGG16, ResNet50 and YOLO. VGG16 and ResNet50 are the pre-trained feature extraction model which are trained on millions of images. YOLO is used for real time object detection. It first extracts the image features using VGG16, ResNet50 and YOLO and concatenate the result in to single file. At last LSTM and BiGRU are used for textual description of the image. Proposed model is evaluated by using BLEU, METEOR and RUGE score.

show abstract

“…These models depend on deep neural networks for producing description of input images, that are considered more precise than those generated by the other two categories of methods. In the paper [13],authors proposed dual graph convolutional networks with transformer and curriculum learning for image captioning. They evaluated results on MS-COCO dataset with achieves BLEU-1 score of 82.2 and a BLEU-2 score of 67.6.…”

Section: Related Literature Reviewmentioning

confidence: 99%

Automatic image caption generation using deep learning

Verma

Yadav

Kumar

et al. 2023

Multimed Tools Appl

View full text Add to dashboard Cite

Image captioning is an interesting and challenging task with applications in diverse domains such as image retrieval, organizing and locating images of users' interest etc. It has huge potential for replacing manual caption generation for images and is especially suitable for large scale image data. Recently, deep neural network based methods have achieved great success in the field of computer vision, machine translation and language generation. In this paper, we propose an encoder-decoder based model that is capable of generating grammatically correct captions for images. This model makes use of VGG16 Hybrid Places 1365 as encoder and LSTM as decoder.To ensure the complete ground truth accuracy, the model is trained on the labelled Flickr8k and MSCOCO Captions datasets. Further, the model is evaluated using all standard metrics such as BLEU, METEOR, GLEU and ROUGE L. Experimental results indicate that the proposed model obtained a BLEU-1 score 0.6666, METEOR score 0.5060 and GLEU score 0.2469 on Flickr8k dataset and BLEU-1 score 0.7350, METEOR score 0.4768 and GLEU score 0.2798 on MSCOCO Captions dataset. Thus, the proposed method achieved a significant performance as compared to the state-of-art approaches. To evaluate the efficacy of the model further, we also show the results of caption generation from live sample images that reinforces the validity of the proposed approach.

show abstract

Dual Graph Convolutional Networks with Transformer and Curriculum Learning for Image Captioning

Cited by 6 publications

References 50 publications

An Extensive Analysis of Image Captioning Models, Evaluation Measures, and Datasets

An Extensive Analysis of Image Captioning Models, Evaluation Measures, and Datasets

An Efficient Deep Learning based Hybrid Model for Image Caption Generation

Automatic image caption generation using deep learning

Contact Info

Product

Resources

About