Pretrained Word Embeddings for Image Captioning

Atliha, Viktar; Šešok, Dmitrij

doi:10.1109/estream53087.2021.9431465

Cited by 5 publications

(4 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition, the previous model (Bi-LSTM G,+M ) [4] Moreover, in recent interesting work [3,4], the authors found that small dataset like Flickr8K which has difficulty to train the deep models with insufficient data. Nonetheless, the proposed Bi-LSTM E,+W model substantially outperforms on all metrics for both word and syllable segmentation tasks although the size of the corpus is small (around 50460 sentences for 10K images), compare with other different models namely GRU E , Bi-GRU E , LSTM E , Bi-LSTM G , Bi-LSTM N , Bi-LSTM E , the baseline models [1,2] as well as the state-of-the-art models [3,4,5].…”

Section: Comparison With State-of-the-art Methodsmentioning

confidence: 99%

“…ResNet101 is used as feature extraction model and Standard LSTM with one cell is utilized as decoder. The pretrained vector representations as Word2Vec and GloVe embedding are compared on the MSCOCO dataset [5]. The model performance with GloVe vectors achieved better results than the model with Word2Vec because image captioning is more suitable with co-occurrence of word pairs in the entire corpus.…”

Section: Related Workmentioning

confidence: 99%

“…We investigated with different dimensions to find which is the best for both word and syllable segmentations and modelling of word vectors for Myanmar language. More than that, different deep learning models such as EfficientNetB7 with GRU, EfficientNetB7 with Bi-GRU, EfficientNetB7 with LSTM, EfficientNetB7 with Bi-LSTM using GloVe embedding model, EfficientNetB7 with Bi-LSTM using one-hot embedding model, VGG16 with Bi-LSTM models and NASNetLarge with Bi-LSTM models are compared with baseline models [1,2] and state-ofthe-art models [3,4,5]. The experimental results showed that the EfficientNetB7 with Bi-LSTM using GloVe embedding vectors can give significantly better performance for both segmentations in Myanmar image captioning compared with other different models.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

EfficientNetB7 and Bi-LSTM with GloVe Vector Based Myanmar Image Captioning

2023

IJIES

View full text Add to dashboard Cite

Image Captioning (IC) is one of the most widely discussed topic in Artificial Intelligence. In this paper, Myanmar image caption is generated using EfficientNetB7 and Bidirectional Long Short-Term Memory (Bi-LSTM) with GloVe embedding and described about the comparative analysis results. For the purpose of achieving better performance, Myanmar image caption corpus is created and annotated over 50k sentences for 10k images, which are based on Flickr8k dataset and 2k images are selected from Flickr30k dataset. Two different types of segmentations such as word and syllable segmentation level are studied in text pre-processing step and then constructed our own GloVe vectors for both segmentations. As far as being aware and up to our knowledge, this is the first attempt of applying syllable and word vector features in neural network-based Myanmar IC system and then compared with one-hot encoding vectors on various different models. According to the evaluations results, EfficientNetB7 with Bi-LSTM using word and syllable GloVe embedding outperforms than EfficientNetB7 and Bi-LSTM with one-hot encoding, other neural networks such as Gated Recurrent Unit (GRU), Bidirectional Gated Recurrent Unit (Bi-GRU), and Long Short-Term Memory (LSTM), VGG16 with Bi-LSTM, NASNetLarge with Bi-LSTM models as well as baseline models. EffecientNetB7 with Bi-LSTM using GloVe vectors achieved the highest BLEU-4 score of 35.09%, 49.52% of ROUGE-L, 54.34% of ROUGE-SU4 and 21.3% of METEOR score on word vectors, and the highest BLEU-4 score of 46.2%, 65.62% of ROUGE-L, 68.43% of ROUGE-SU4 and 27.07% of METEOR score on syllable vectors.

show abstract

Section: Comparison With State-of-the-art Methodsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

EfficientNetB7 and Bi-LSTM with GloVe Vector Based Myanmar Image Captioning

2023

IJIES

View full text Add to dashboard Cite

show abstract

“…Other studies, such as Vinyals et al [18], have found that using pre-trained embedding does not improve model performance. Atliha et al [19] used pre-trained embedding, which is GloVe, with fine-tuning and found that it can improve the performance of the model and are most suitable for the image captioning model training improvement.…”

Section: Lstm Ms Cocomentioning

confidence: 99%

Enhanced Image Captioning Using Features Concatenation and Efficient Pre-Trained Word Embedding

Elbedwehy¹,

Medhat²,

Hamza³

et al. 2023

Computer Systems Science and Engineering

View full text Add to dashboard Cite

One of the issues in Computer Vision is the automatic development of descriptions for images, sometimes known as image captioning. Deep Learning techniques have made significant progress in this area. The typical architecture of image captioning systems consists mainly of an image feature extractor subsystem followed by a caption generation lingual subsystem. This paper aims to find optimized models for these two subsystems. For the image feature extraction subsystem, the research tested eight different concatenations of pairs of vision models to get among them the most expressive extracted feature vector of the image. For the caption generation lingual subsystem, this paper tested three different pre-trained language embedding models: Glove (Global Vectors for Word Representation), BERT (Bidirectional Encoder Representations from Transformers), and TaCL (Token-aware Contrastive Learning), to select from them the most accurate pre-trained language embedding model. Our experiments showed that building an image captioning system that uses a concatenation of the two Transformer based models SWIN (Shifted window) and PVT (Pyramid Vision Transformer) as an image feature extractor, combined with the TaCL language embedding model is the best result among the other combinations.

show abstract