Show, Control and Tell: A Framework for Generating Controllable and Grounded Captions

Cornia, Marcella; Baraldi, Lorenzo; Cucchiara, Rita

doi:10.1109/cvpr.2019.00850

Cited by 191 publications

(162 citation statements)

References 47 publications

Supporting

Mentioning

162

Contrasting

Order By: Relevance

“…For example, we could learn a ranker to help us choose the best contents (Stent et al, 2004). Or we could manually define some matching rules to help rank the selection (Cornia et al, 2018). In Table 2, we show the VRS model achieves very high metric scores based on an oracle ranker, so learning a ranker should be able to improve the performance straightforwardly.…”

Section: A Performance/controllability Trade-offmentioning

confidence: 99%

Select and Attend: Towards Controllable Content Selection in Text Generation

Shen¹,

Suzuki

Inui

et al. 2019

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conferen

View full text Add to dashboard Cite

Many text generation tasks naturally contain two steps: content selection and surface realization. Current neural encoder-decoder models conflate both steps into a black-box architecture. As a result, the content to be described in the text cannot be explicitly controlled. This paper tackles this problem by decoupling content selection from the decoder. The decoupled content selection is human interpretable, whose value can be manually manipulated to control the content of generated text. The model can be trained end-to-end without human annotations by maximizing a lower bound of the marginal likelihood. We further propose an effective way to trade-off between performance and controllability with a single adjustable hyperparameter. In both data-to-text and headline generation tasks, our model achieves promising results, paving the way for controllable content selection in text generation.

show abstract

Section: A Performance/controllability Trade-offmentioning

confidence: 99%

Select and Attend: Towards Controllable Content Selection in Text Generation

Shen¹,

Suzuki

Inui

et al. 2019

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conferen

View full text Add to dashboard Cite

show abstract

“…Lu et al [15] first generate a sentence template with blank slots which will be filled in by visual concepts using object detectors. Cornia et al [16] propose a controllable approach to shift the rank of salient image regions by shift gate with adaptive attention. Li et al [17] introduce a new architecture to facilitate vocabulary expansion and produce novel objects via pointing mechanism and object learners.…”

Section: Related Workmentioning

confidence: 99%

A Sparse Transformer-Based Approach for Image Captioning

Zhou

Chen

et al. 2020

IEEE Access

View full text Add to dashboard Cite

Image Captioning is the task of providing a natural language description for an image. It has caught significant amounts of attention from both computer vision and natural language processing communities. Most image captioning models adopt deep encoder-decoder architectures to achieve state-of-the-art performances. However, it is difficult to model knowledge on relationships between input image region pairs in the encoder. Furthermore, the word in the decoder hardly knows the correlation to specific image regions. In this paper, a novel deep encoder-decoder model is proposed for image captioning which is developed on sparse Transformer framework. The encoder adopts a multi-level representation of image features based on self-attention to exploit low-level and high-level features, naturally the correlations between image region pairs are adequately modeled as self-attention operation can be seen as a way of encoding pairwise relationships. The decoder improves the concentration of multi-head self-attention on the global context by explicitly selecting the most relevant segments at each row of the attention matrix. It can help the model focus on the more contributing image regions and generate more accurate words in the context. Experiments demonstrate that our model outperforms previous methods and achieves higher performance on MSCOCO and Flickr30k datasets. Our code is available at https://github.com/2014gaokao/ImageCaptioning. INDEX TERMS Image captioning, self-attention, explict sparse, local adaptive threshold.

show abstract

“…The authors declare no conflict of interest. [20] x x Visual Genome Dataset [21] x x x 19.9 13.7 13.1 [22] x x x x [23] x [35] x x Recall Evaluation metric [36] x x OI, VG, VRD [37] x x X X 71.6 51.8 37.1 26.5 24.3 [38] x x APRC, CSMC [39] x x x x F-1 score metrics 21.6 [45] x x x x IAPRTC-12 [46] x x x x [47] x x x x x R [48] x…”

Section: Conflicts Of Interestmentioning

confidence: 99%

A Systematic Literature Review on Image Captioning

Staniūtė

Šešok

2019

Applied Sciences

View full text Add to dashboard Cite

Natural language problems have already been investigated for around five years. Recent progress in artificial intelligence (AI) has greatly improved the performance of models. However, the results are still not sufficiently satisfying. Machines cannot imitate human brains and the way they communicate, so it remains an ongoing task. Due to the increasing amount of information on this topic, it is very difficult to keep on track with the newest researches and results achieved in the image captioning field. In this study a comprehensive Systematic Literature Review (SLR) provides a brief overview of improvements in image captioning over the last four years. The main focus of the paper is to explain the most common techniques and the biggest challenges in image captioning and to summarize the results from the newest papers. Inconsistent comparison of results achieved in image captioning was noticed during this study and hence the awareness of incomplete data collection is raised in this paper. Therefore, it is very important to compare results of a newly created model produced with the newest information and not only with the state of the art methods. This SLR is a source of such information for researchers in order for them to be precisely correct on result comparison before publishing new achievements in the image caption generation field.

show abstract

Show, Control and Tell: A Framework for Generating Controllable and Grounded Captions

Cited by 191 publications

References 47 publications

Select and Attend: Towards Controllable Content Selection in Text Generation

Select and Attend: Towards Controllable Content Selection in Text Generation

A Sparse Transformer-Based Approach for Image Captioning

A Systematic Literature Review on Image Captioning

Contact Info

Product

Resources

About