Semi-Supervised Image Captioning by Adversarially Propagating Labeled Data

Kim, Dong-Jin; Oh, Tae-Hyun; Choi, Jinsoo; Kweon, In So

doi:10.48550/arxiv.2301.11174

Cited by 1 publication

(2 citation statements)

References 57 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To verify the effectiveness of our proposed model, we conduct tests on the MOCS test set and compare the results with those of other models. As presented in Table 1, we first calculate the metrics of BLEU@1, BLEU@4, METEOR, ROUGE_L, CIDEr, and SPICE based on cross-entropy loss using Equation (20), with the corresponding results shown in columns 2 through 7 of Table 1. Next, we calculate the metrics based on CIDEr-D optimization using Equation (21), and the corresponding results are shown in columns 8 through 13 of Table 1.…”

Section: Experimental Results 641 Experimental Results On Mocsmentioning

confidence: 99%

“…In terms of model architecture, previous approaches to image captioning have predominantly relied on convolutional neural networks (CNN) and long short-term memory networks (LSTM). CNN is used for encoding spatial features, while LSTM decodes these features into textual descriptions, achieving certain success in image-captioning tasks [ 18 , 19 , 20 ] on public datasets. However, these architectures are limited by the expressive power and training efficiency of LSTM and cannot further improve their performance.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Style-Enhanced Transformer for Image Captioning in Construction Scenes

Song,

Chen,

Wang

2024

Entropy

View full text Add to dashboard Cite

Image captioning is important for improving the intelligence of construction projects and assisting managers in mastering construction site activities. However, there are few image-captioning models for construction scenes at present, and the existing methods do not perform well in complex construction scenes. According to the characteristics of construction scenes, we label a text description dataset based on the MOCS dataset and propose a style-enhanced Transformer for image captioning in construction scenes, simply called SETCAP. Specifically, we extract the grid features using the Swin Transformer. Then, to enhance the style information, we not only use the grid features as the initial detail semantic features but also extract style information by style encoder. In addition, in the decoder, we integrate the style information into the text features. The interaction between the image semantic information and the text features is carried out to generate content-appropriate sentences word by word. Finally, we add the sentence style loss into the total loss function to make the style of generated sentences closer to the training set. The experimental results show that the proposed method achieves encouraging results on both the MSCOCO and the MOCS datasets. In particular, SETCAP outperforms state-of-the-art methods by 4.2% CIDEr scores on the MOCS dataset and 3.9% CIDEr scores on the MSCOCO dataset, respectively.

show abstract

Section: Experimental Results 641 Experimental Results On Mocsmentioning

confidence: 99%