An Attentive Fourier-Augmented Image-Captioning Transformer

Raymond, Osolo Ian; Yang, Zhan; Long, Jun

doi:10.3390/app11188354

Cited by 5 publications

(3 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The encoder-decoder paradigm has been proposed whereas a global-enhanced encoder first encodes the original inputs into highly abstract localised representations and then extracts the intra-and inter-layer global representation. The decoder then implements the suggested global adaptive controller to iteratively incorporate the multimodal information while producing the caption word by word [52]. An Attentive Fourier-Augmented Image Captioning Transformer (AFCT) based methodology has been proposed by the researchers.…”

Section: Analysis Using State-of-the-art Methodsmentioning

confidence: 99%

Dual-Modal Transformer with Enhanced Inter- and Intra-Modality Interactions for Image Captioning

et al. 2022

View full text Add to dashboard Cite

Image captioning is oriented towards describing an image with the best possible use of words that can provide a semantic, relatable meaning of the scenario inscribed. Different models can be used to accomplish this arduous task depending on the context and requirement of what needs to be achieved. An encoder–decoder model which uses the image feature vectors as an input to the encoder is often marked as one of the appropriate models to accomplish the captioning process. In the proposed work, a dual-modal transformer has been used which captures the intra- and inter-model interactions in a simultaneous manner within an attention block. The transformer architecture is quantitatively evaluated on a publicly available Microsoft Common Objects in Context (MS COCO) dataset yielding a Bilingual Evaluation Understudy (BLEU)-4 Score of 85.01. The efficacy of the model is evaluated on Flickr 8k, Flickr 30k datasets and MS COCO datasets and results for the same is compared and analysed with the state-of-the-art methods. The results shows that the proposed model outperformed when compared with conventional models, such as the encoder–decoder model and attention model.

show abstract

Section: Analysis Using State-of-the-art Methodsmentioning

confidence: 99%

Dual-Modal Transformer with Enhanced Inter- and Intra-Modality Interactions for Image Captioning

et al. 2022

View full text Add to dashboard Cite

show abstract

“…It depends on the attention mechanism, specifically self-attention. Osolo et al [80] applied fast Fourier transforms to decompose the input features and extract more important information from the images to provide succinct and informative captions. To distinguish between the word semantics and grammatical structures of captions and include the PoS guiding information in the modeling, Wang et al [81] proposed a novel part-of-speech guided transformer (PoS-Transformer).…”

Section: Transformermentioning

confidence: 99%

A Systematic Literature Review on Using the Encoder-Decoder Models for Image Captioning in English and Arabic Languages

Alsayed,

Arif,

Qadah

et al. 2023

Applied Sciences

View full text Add to dashboard Cite

With the explosion of visual content on the Internet, creating captions for images has become a necessary task and an exciting topic for many researchers. Furthermore, image captioning is becoming increasingly important as the number of people utilizing social media platforms grows. While there is extensive research on English image captioning (EIC), studies focusing on image captioning in other languages, especially Arabic, are limited. There has also yet to be an attempt to survey Arabic image captioning (AIC) systematically. This research aims to systematically survey encoder-decoder EIC while considering the following aspects: visual model, language model, loss functions, datasets, evaluation metrics, model comparison, and adaptability to the Arabic language. A systematic review of the literature on EIC and AIC approaches published in the past nine years (2015–2023) from well-known databases (Google Scholar, ScienceDirect, IEEE Xplore) is undertaken. We have identified 52 primary English and Arabic studies relevant to our objectives (The number of articles on Arabic captioning is 11, and the rest are for the English language). The literature review shows that applying the English-specific models to the Arabic language is possible, with the use of a high-quality Arabic database and following the appropriate preprocessing. Moreover, we discuss some limitations and ideas to solve them as a future direction.

show abstract

“…As a result, deep learning models have been getting more complicated, bigger and a lot of the state-of-the-art performance of a number of models such as GPT, BERT can be attributed to the amount data used to train them. Some methods using simpler architectures have been proposed to reverse this trend such as using Fourier transforms and fully connected layers [7,33] to get performances close to the state-of-the-art.…”

Section: Image Captioningmentioning

confidence: 99%

An Analysis of the Use of Feed-Forward Sub-Modules for Transformer-Based Image Captioning Tasks

2021

Self Cite

View full text Add to dashboard Cite

In the quest to make deep learning systems more capable, a number of more complex, more computationally expensive and memory intensive algorithms have been proposed. This switchover glosses over the capabilities of many of the simpler systems or modules within them to adequately address current and future problems. This has led to some of the deep learning research being inaccessible to researchers who don’t possess top-of-the-line hardware. The use of simple feed forward networks has not been explicitly explored in the current transformer-based vision-language field. In this paper, we use a series of feed-forward layers to encode image features, and caption embeddings, alleviating some of the effects of the computational complexities that accompany the use of the self-attention mechanism and limit its application in long sequence task scenarios. We demonstrate that a decoder does not require masking for conditional short sequence generation where the task is not only dependent on the previously generated sequence, but another input such as image features. We perform an empirical and qualitative analysis of the use of linear transforms in place of self-attention layers in vision-language models, and obtain competitive results on the MSCOCO dataset. Our best feed-forward model obtains average scores of over 90% of the current state-of-the-art pre-trained Oscar model in the conventional image captioning metrics. We also demonstrate that the proposed models take less time training and use less memory at larger batch sizes and longer sequence lengths.

show abstract

An Attentive Fourier-Augmented Image-Captioning Transformer

Cited by 5 publications

References 37 publications

Dual-Modal Transformer with Enhanced Inter- and Intra-Modality Interactions for Image Captioning

Dual-Modal Transformer with Enhanced Inter- and Intra-Modality Interactions for Image Captioning

A Systematic Literature Review on Using the Encoder-Decoder Models for Image Captioning in English and Arabic Languages

An Analysis of the Use of Feed-Forward Sub-Modules for Transformer-Based Image Captioning Tasks

Contact Info

Product

Resources

About