Where to put the image in an image caption generator

Abstract

When a recurrent neural network language model is used for caption generation, the image information can be fed to the neural network either by directly incorporating it in the RNN -conditioning the language model by 'injecting' image features -or in a layer following the RNNconditioning the language model by 'merging' image features. While both options are attested in the literature, there is as yet no systematic comparison between the two. In this paper we empirically show that it is not especially detrimental to performance whether one architecture is used or another. The merge architecture does have practical advantages, as conditioning by merging allows the RNN's hidden state vector to shrink in size by up to four times. Our results suggest that the visual and linguistic modalities for caption generation need not be jointly encoded by the RNN as that yields large, memory-intensive models with few tangible advantages in performance; rather, the multimodal integration should be delayed to a subsequent stage.

Pre-gen Metrics: Predicting Caption Quality Metrics Without Generating Captions

“…For our experiments, we used a variety of pre-trained neural caption generators (36 in all) from [23]. 4 These models are based on four different caption generator architectures.…”

What goes into a word: generating image descriptions with top-down spatial knowledge

“…Our method is to add step-by-step modules and configurations to the network providing different kind of top-down knowledge in Section 2 and investigating the performance of such configura-tions. There are several design choices with small effects on the performance but costly in terms of parameter size (Tanti et al, 2018b). Therefore, if there is no research question related to that choice, we take the simplest choice as reported in the previous work such as (Lu et al, 2017;Anderson et al, 2018).…”

English to Hindi Multi-modal Neural Machine Translation and Hindi Image Captioning

“…The multimodal NMT toolkit is employed to build the multimodal NMT system for multimodal translation task, which are based on the pytorch port of OpenNMT (Klein et al, 2017). For text-only translation task, OpenNMT is deployed to build the NMT system and in the case of Hindi-only image captioning track, publicly available VGG16 and LSTM in Keras library, are used to build the system (Simonyan and Zisserman, 2015;Tanti et al, 2018). We have used Hindi visual genome dataset in each track of WAT2019 multi-modal translation task provided by the organizer (Nakazawa et al, 2019).…”

“…Hence, we have chosen predicted translation at an optimum point on 24,000 epoch. In the training process of Hindi-only image captioning track, we have used merge-model following settings of (Tanti et al, 2018). The preprocessed image feature vector of 4096 elements are processed by a dense layer to provide 256 elements for representation of the image.…”