Length-Controllable Image Captioning

Deng, Chaorui; Ding, Ning; Tan, Mingkui; Wu, Qi

doi:10.1007/978-3-030-58601-0_42

Cited by 52 publications

(55 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The purpose of this experiment for various τ is to simulate the trace drawing speed of users in a real application scenario, and a larger τ is equivalent to a faster drawing speed. As Deng et al (2020) has demonstrated, the length is one of the critical facts that impact quantitative performance. This result implies we can further decide to generate either a coarse-grained or fine-grained caption by controlling the time-frequency τ .…”

Section: Quantitative Analysis Controllability Analysis On Temporal Ordermentioning

confidence: 99%

“…Controllable Image Captioning is an emerging research direction. Previous works aim to control the captioning by Part-Of-Speech tagging (Deshpande et al, 2018), sentiment, (You et al, 2018), length (Deng et al, 2020), bounding box (Cornia et al, 2019) etc. Those works either tried to describe a semantic guided captioning.…”

Section: Related Workmentioning

confidence: 99%

“…To generate controllable and explainable captions, recent works dedicated to establishing a new controllable image captioning task to generate the caption at will. The captioning process can be controlled by POS tagging (Deshpande et al, 2018), sentiment (You et al, 2018), length (Deng et al, 2020), bounding boxes (Cornia et al, 2019), and mouse traces (Pont-Tuset et al, 2020).…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Control Image Captioning Spatially and Temporally

Yan¹,

Ji²,

Luo³

et al. 2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

View full text Add to dashboard Cite

Generating image captions with user intention is an emerging need. The recently published Localized Narratives dataset takes mouse traces as another input to the image captioning task, which is an intuitive and efficient way for a user to control what to describe in the image. However, how to effectively employ traces to improve generation quality and controllability is still under exploration. This paper aims to solve this problem by proposing a novel model called LoopCAG, which connects Contrastive constraints and Attention Guidance in a Loop manner, engaged explicit spatial and temporal constraints to the generating process. Precisely, each generated sentence is temporally aligned to the corresponding trace sequence through a contrastive learning strategy. Besides, each generated text token is supervised to attend to the correct visual objects under heuristic spatial attention guidance. Comprehensive experimental results demonstrate that our LoopCAG model learns better correspondence among the three modalities(vision, language, and traces) and achieves SOTA performance on trace controlled image captioning task. Moreover, the controllability and explainability of LoopCAG are validated by analyzing spatial and temporal sensitivity during the generation process.

show abstract

Section: Quantitative Analysis Controllability Analysis On Temporal Ordermentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Control Image Captioning Spatially and Temporally

Yan¹,

Ji²,

Luo³

et al. 2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

View full text Add to dashboard Cite

show abstract

“…During training, we sample a masking ratio from a uniform prior distribution ([0,1]) and randomly mask the percentage of target tokens for prediction. While in inference, we adopt a non-autoregressive sampling strategy (i.e., mask-predict-k strategy [2,3,5]). So only a few sampling steps (e.g., 4) are needed to generate all target tokens, which enable real-time inference.…”

Section: Training and Inference Strategymentioning

confidence: 99%

A Picture is Worth a Thousand Words

Huang

Liu

et al. 2021

Proceedings of the 29th ACM International Conference on Multimedia

View full text Add to dashboard Cite

A creative image-and-text generative AI system mimics humans' extraordinary abilities to provide users with diverse and comprehensive caption suggestions, as well as rich image creations. In this work, we demonstrate such an AI creation system to produce both diverse captions and rich images. When users imagine an image and associate it with multiple captions, our system paints a rich image to reflect all captions faithfully. Likewise, when users upload an image, our system depicts it with multiple diverse captions. We propose a unified multi-modal framework to achieve this goal. Specifically, our framework jointly models image-and-text representations with a Transformer network, which supports rich image creation by accepting multiple captions as input. We consider the relations among input captions to encourage diversity in training and adopt a non-autoregressive decoding strategy to enable realtime inference. Based on these, our system supports both diverse captions and rich images generations. Our code is available online 1 . CCS CONCEPTS• Computing methodologies → Natural language generation; Image processing.

show abstract

“…However, significantly different from them, we borrow the idea from the mask-predict framework (Gu et al, 2018;Ghazvininejad et al, 2019;Deng et al, 2020) to progressively incorporate pairwise ordering information into SE-Graph, which is the basis of our graph-based sentence ordering model. To the best of our knowledge, our work is the first attempt to explore iteratively refined GNN for sentence ordering.…”

Section: Related Workmentioning

confidence: 99%

Improving Graph-based Sentence Ordering with Iteratively Predicted Pairwise Orderings

Lai¹,

Wang²,

Meng³

et al. 2021

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Dominant sentence ordering models can be classified into pairwise ordering models and set-to-sequence models. However, there is little attempt to combine these two types of models, which inituitively possess complementary advantages. In this paper, we propose a novel sentence ordering framework which introduces two classifiers to make better use of pairwise orderings for graph-based sentence ordering (Yin et al., 2019(Yin et al., , 2021. Specially, given an initial sentence-entity graph, we first introduce a graph-based classifier to predict pairwise orderings between linked sentences. Then, in an iterative manner, based on the graph updated by previously predicted highconfident pairwise orderings, another classifier is used to predict the remaining uncertain pairwise orderings. At last, we adapt a GRN-based sentence ordering model (Yin et al., 2019(Yin et al., , 2021 on the basis of final graph. Experiments on five commonly-used datasets demonstrate the effectiveness and generality of our model. Particularly, when equipped with BERT (Devlin et al., 2019) and FHDecoder (Yin et al., 2020), our model achieves state-of-the-art performance. Our code is available at https:// github.com/DeepLearnXMU/IRSEG.

show abstract

Length-Controllable Image Captioning

Cited by 52 publications

References 35 publications

Control Image Captioning Spatially and Temporally

Control Image Captioning Spatially and Temporally

A Picture is Worth a Thousand Words

Improving Graph-based Sentence Ordering with Iteratively Predicted Pairwise Orderings

Contact Info

Product

Resources

About