Comprehensive Image Captioning via Scene Graph Decomposition

Abstract

We address the challenging problem of image captioning by revisiting the representation of image scene graph. At the core of our method lies the decomposition of a scene graph into a set of subgraphs, with each sub-graph capturing a semantic component of the input image. We design a deep model to select important sub-graphs, and to decode each selected sub-graph into a single target sentence. By using sub-graphs, our model is able to attend to different components of the image. Our method thus accounts for accurate, diverse, grounded and controllable captioning at the same time. We present extensive experiments to demonstrate the benefits of our comprehensive captioning model. Our method establishes new state-of-the-art results in caption diversity, grounding, and controllability, and compares favourably to latest methods in caption quality. Our project website can be found at http://pages.cs.wisc.edu/ ~yiwuzhong/Sub-GC.html.

Probabilistic Modeling of Semantic Ambiguity for Scene Graph Generation

“…‡ Yong Zhang, Baoyuan Wu and Yujiu Yang are the corresponding authors. as image captioning [33,42] and visual question answering [36]. Intuitively, the latter would get greater benefit from more human-like scene graphs.…”