Analysis of diversity-accuracy tradeoff in image captioning

Luo, Ruotian; Shakhnarovich, Gregory

doi:10.48550/arxiv.2002.11848

Cited by 6 publications

(6 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Using top-K sampling decreases the top-1 accuracy yet significantly increases all diversity scores (-3.6 CIDEr yet +22% in 1-gram with T=0.6, K=3). The same trend was also observed in [36].…”

Section: Ablation Studysupporting

confidence: 84%

See 1 more Smart Citation

Comprehensive Image Captioning via Scene Graph Decomposition

Zhong

Wang

Chen

et al. 2020

Preprint

View full text Add to dashboard Cite

We address the challenging problem of image captioning by revisiting the representation of image scene graph. At the core of our method lies the decomposition of a scene graph into a set of subgraphs, with each sub-graph capturing a semantic component of the input image. We design a deep model to select important sub-graphs, and to decode each selected sub-graph into a single target sentence. By using sub-graphs, our model is able to attend to different components of the image. Our method thus accounts for accurate, diverse, grounded and controllable captioning at the same time. We present extensive experiments to demonstrate the benefits of our comprehensive captioning model. Our method establishes new state-of-the-art results in caption diversity, grounding, and controllability, and compares favourably to latest methods in caption quality. Our project website can be found at http://pages.cs.wisc.edu/ ~yiwuzhong/Sub-GC.html.

show abstract

Section: Ablation Studysupporting

confidence: 84%

“…After NMS, top-ranked sub-graphs are decoded using an attention-based LSTM. As shown in [36], an optional top-K sampling [15,43] can be applied during the decoding to further improve caption diversity. We disable top-K sampling for our experiments unless otherwise noticed.…”

Section: Training and Inferencementioning

confidence: 99%

Comprehensive Image Captioning via Scene Graph Decomposition

Zhong

Wang

Chen

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…[28] is another exploration of Variational structure, which proposed a novel variational multi-modal inferring tree (similar to the syntax tree) to improve the lexical and syntactic diversity in captioning. At last, [4,1] are two works which also concerned about the relation between accuracy and diversity like the main purpose of this work. [1] proposed an off-policy strategy to increase the range of samples during RL training, which improves the diversity dramatically, however, also causes the same dramatic decrease of accuracy.…”

Section: Related Workmentioning

confidence: 99%

“…Diversity metrics For diversity evaluation, we adopt five benchmark diversity metrics in [27,28,4,1]. 1) n-gram diversity (Div-n): the ratio of distinct n-grams to the total number of words in the generated captions.…”

Section: Accuracy Metricsmentioning

confidence: 99%

“…However, for machine-learning-based works, it is hard to reveal the true semantic performance with low accuracy. On the other hand, common metrics proposed to capture diversity involve no accuracy mea-sure [4], thus, can be misleading with wrong words to manifest inflating diversity scores. Therefore, we suppose that the diverse generation should be dependent on the accuracy evaluation.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Variational Transformer: A Framework Beyond the Trade-off between Accuracy and Diversity for Image Captioning

Yang¹,

Shang²,

Liu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Accuracy and Diversity are two essential metrizable manifestations in generating natural and semantically correct captions. Many efforts have been made to enhance one of them with another decayed due to the trade-off gap. However, compromise does not make the progress. Decayed diversity makes the captioner a repeater, and decayed accuracy makes it a fake advisor. In this work, we exploit a novel Variational Transformer framework to improve accuracy and diversity simultaneously. To ensure accuracy, we introduce the "Invisible Information Prior" along with the "Auto-selectable GMM" to instruct the encoder to learn the precise language information and object relation in different scenes. To ensure diversity, we propose the "Range-Median Reward" baseline to retain more diverse candidates with higher rewards during the RL-based training process. Experiments show that our method achieves the simultaneous promotion of accuracy (CIDEr) and diversity (self-CIDEr), up to 1.1 and 4.8 percent, compared with the baseline. Also, our method outperforms others under the newly proposed measurement of the trade-off gap, with at least 3.55 percent promotion.Preprint. Under review.

show abstract