Semi-Autoregressive Image Captioning

Abstract

Current state-of-the-art approaches for image captioning typically adopt an autoregressive manner, i.e., generating descriptions word by word, which suffers from slow decoding issue and becomes a bottleneck in real-time applications. Non-autoregressive image captioning with continuous iterative refinement, which eliminates the sequential dependence in a sentence generation, can achieve comparable performance to the autoregressive counterparts with a considerable acceleration. Nevertheless, based on a well-designed experiment, we empirically proved that iteration times can be effectively reduced when providing sufficient prior knowledge for the language decoder. Towards that end, we propose a novel two-stage framework, referred to as Semi-Autoregressive Image Captioning (SAIC), to make a better trade-off between performance and speed. The proposed SAIC model maintains autoregressive property in global but relieves it in local. Specifically, SAIC model first jumpily generates an intermittent sequence in an autoregressive manner, that is, it predicts the first word in every word group in order. Then, with the help of the partially deterministic prior information and image features, SAIC model non-autoregressively fills all the skipped words with one iteration. Experimental results on the MS COCO benchmark demonstrate that our SAIC model outperforms the preceding non-autoregressive image captioning models while obtaining a competitive inference speedup. Code is available at https://github.com/feizc/SAIC.

Attention-Aligned Transformer for Image Captioning

“…The attention mechanism is first introduced to augment vanilla recurrent network (Bahdanau, Cho, and Bengio 2014;Luong, Pham, and Manning 2015) (Li et al 2019;Fei 2019;Cornia et al 2020;Pan et al 2020;Fei 2021;Yan et al 2021;Ji et al 2021) are proposed to replace conventional RNN, achieving new state-of-the-art performances. However, as far as we concerned, improving the attention distribution with self-supervised mask perturbation has never been studied in image captioning task, which push forward our exploration in this paper.…”

Uncertainty-Aware Image Captioning

“…Besides, (Fei 2020) introduces latent variables to eliminate the modal gap and develop a more powerful probabilistic framework to simulate more complicated distributions. (Yan et al 2021) splits the captions into word groups averagely and produces the group synchronously. In addition to parallel generation, a range of semi-autoregressive models (Wang, Zhang, and Chen 2018;Ghazvininejad, Levy, and Zettlemoyer 2020;Stern et al 2019;Gu, Wang, and Zhao 2019;Fei 2021b;Fei et al 2022b,a;Zhou et al 2021) pay attention to nonmonotonic sequence generation with limited forms of autoregressiveness, i.e., tree-like traversal, which are mainly based on the insertion operation.…”