Multilevel attention and relation network based image captioning model

Sharma, Himanshu; Srivastava, Swati

doi:10.1007/s11042-022-13793-0

Cited by 15 publications

(4 citation statements)

References 64 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To accomplish this objective, the author presents a Locality-Sensitive Transformer Network (LSTNet) with two novel designs Locality-Sensitive Attention and Locality-Sensitive Fusion (LSF). In [29], a Local Relation Network (LRN) was planned over the objects and image regions that not only determines the connection among the object and image regions among them creates major context-based features equivalent to all the regions from the image. Lastly, a different typical LSTM utilizes an attention process that concentrates on related contextual data, spatial places, and deep visual features.…”

Section: Related Workmentioning

confidence: 99%

Lighting Search Algorithm With Convolutional Neural Network-Based Image Captioning System for Natural Language Processing

Alnashwan,

Chelloug,

Almalki

et al. 2023

IEEE Access

View full text Add to dashboard Cite

Recently, deep learning models have become more prominent due to their tremendous performance for real-time tasks like face recognition, object detection, natural language processing (NLP), instance segmentation, image classification, gesture recognition, and video classification. Image captioning is one of the critical tasks in NLP and computer vision (CV). It completes conversion from image to text; specifically, the model produces description text automatically based on the input images. In this aspect, this article develops a Lighting Search Algorithm (LSA) with a Hybrid Convolutional Neural Network Image Captioning System (LSAHCNN-ICS) for NLP. This introduced LSAHCNN-ICS system develops an end-toend model which employs convolutional neural network (CNN) based ShuffleNet as an encoder and HCNN as a decoder. At the encoding part, the ShuffleNet model derives feature descriptors of the image. Besides, in the decoding part, the description of text can be generated using the proposed hybrid convolutional neural network (HCNN) model. To achieve improved captioning results, the LSA is applied as a hyperparameter tuning strategy, representing the innovation of the study. The simulation analysis of the presented LSAHCNN-ICS technique is performed on a benchmark database, and the obtained results demonstrated the enhanced outcomes of the LSAHCNN-ICS algorithm over other recent methods with maximum Consensusbased Image Description Evaluation (CIDEr Code) of 43.60, 59.54, and 135.14 on Flickr8k, Flickr30k, and MSCOCO datasets correspondingly.

show abstract

Section: Related Workmentioning

confidence: 99%

Lighting Search Algorithm With Convolutional Neural Network-Based Image Captioning System for Natural Language Processing

Alnashwan,

Chelloug,

Almalki

et al. 2023

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Cornia et al [27] extend selfattention with additional "slots" to encode prior information. Sharma et al [29] design an LRN that discovers the relationship between the object and the image regions. Apart from this, BERT, a pre-trained model composed of deep bidirectional transformers, achieved outstanding results.…”

Section: Image Captioningmentioning

confidence: 99%

Novel Object Captioning with Semantic Match from External Knowledge

Zhu

Lin

et al. 2023

Applied Sciences

View full text Add to dashboard Cite

Automatically describing the content of an image is a challenging task that is on the edge between natural language and computer vision. The current image caption models can describe the objects that are frequently seen in the training set very well, but they fail to describe the novel objects that are rarely seen or never seen in the training set. Despite describing novel objects being important for practical applications, only a few works investigate this issue. Furthermore, those works only investigate rarely seen objects, but ignore the never-seen objects. Meanwhile, the number of never-seen objects is more than the number of frequently seen and rarely seen objects. In this paper, we propose two blocks that incorporate external knowledge into the captioning model to solve this issue. Initially, in the encoding phase, the Semi-Fixed Word Embedding block is an improvement for the word embedding layer that enables the captioning model to understand the meaning of the arbitrary visual words rather than a fixed number of words. Furthermore, the Candidate Sentences Selection block chooses candidate sentences by semantic matching rather than probability, avoiding the influence of never-seen words. In experiments, we qualitatively analyze the proposed blocks and quantitatively evaluate several captioning models with the proposed blocks on the Nocaps dataset. The experimental results show the effectiveness of the proposed blocks for novel objects, especially when describing never-seen objects, CIDEr and SPICE improved by 13.1% and 12.0%, respectively.

show abstract

“…An important factor in vision‐to‐language issues like visual question answering and image captioning is the existence of textual information in real‐world images, such as signs, descriptions, and promotional materials 5–9 . However, existing VQA methods 10–14 struggle to comprehend scene text, leading to a decline in performance when answering text‐related questions.…”

Section: Introductionmentioning

confidence: 99%

“…An important factor in vision-to-language issues like visual question answering and image captioning is the existence of textual information in real-world images, such as signs, descriptions, and promotional materials. [5][6][7][8][9] However, existing VQA methods [10][11][12][13][14] struggle to comprehend scene text, leading to a decline in performance when answering text-related questions. In order to efficiently address scene-text based questions, a VQA method must have the capability to recognize and understand scene-text, as well as reason based on the recognized scene-text.…”

Section: Introductionmentioning

confidence: 99%

Enhancing scene‐text visual question answering with relational reasoning, attention and dynamic vocabulary integration

Agrawal,

Jalal,

Sharma

2024

Computational Intelligence

Self Cite

View full text Add to dashboard Cite

Visual question answering (VQA) is a challenging task in computer vision. Recently, there has been a growing interest in text‐based VQA tasks, emphasizing the important role of textual information for better understanding of images. Effectively utilizing text information within the image is crucial for achieving success in this task. However, existing approaches often overlook the contextual information and neglect to utilize the relationships between scene‐text tokens and image objects. They simply incorporate the scene‐text tokens mined from the image into the VQA model without considering these important factors. In this paper, the proposed model initially analyzes the image to extract text and identify scene objects. It then comprehends the question and mines relationships among the question, OCRed text, and scene objects, ultimately generating an answer through relational reasoning by conducting semantic and positional attention. Our decoder with attention map loss enables prediction of complex answers and handles dynamic vocabularies, reducing decoding space. It outperforms softmax‐based cross entropy loss in accuracy and efficiency by accommodating varying vocabulary sizes. We evaluated our model's performance on the TextVQA dataset and achieved an accuracy of 53.91% on the validation set and 53.98% on the test set. Moreover, on the ST‐VQA dataset, our model obtained ANLS scores of 0.699 on the validation set and 0.692 on the test set.

show abstract

Multilevel attention and relation network based image captioning model

Cited by 15 publications

References 64 publications

Lighting Search Algorithm With Convolutional Neural Network-Based Image Captioning System for Natural Language Processing

Lighting Search Algorithm With Convolutional Neural Network-Based Image Captioning System for Natural Language Processing

Novel Object Captioning with Semantic Match from External Knowledge

Enhancing scene‐text visual question answering with relational reasoning, attention and dynamic vocabulary integration

Contact Info

Product

Resources

About