Improving Intra- and Inter-Modality Visual Relation for Image Captioning

Wang, Yong; Zhang, WenKai; Liu, Qing; Zhang, Zhengyuan; Gao, Xin; Sun, Xian

doi:10.1145/3394171.3413877

Cited by 18 publications

(8 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Object relationship has been employed in a few computer vision tasks such as scene graph generation [32]. It encodes the interplay between the object instances to achieve highlevel image understanding to facilitate applications such as image capturing [33] and manipulation [34]. However, such kind of high level relationship is too general and lacks of discrimination for the low-level camera relocalization task at hand.…”

Section: B Object Related Methodsmentioning

confidence: 99%

Objects Matter: Learning Object Relation Graph for Robust Camera Relocalization

Qiao¹,

Xiang²,

Wang³

2022

Preprint

View full text Add to dashboard Cite

Visual relocalization aims to estimate the pose of a camera from one or more images. In recent years deep learning based pose regression methods have attracted many attentions. They feature predicting the absolute poses without relying on any prior built maps or stored images, making the relocalization very efficient. However, robust relocalization under environments with complex appearance changes and real dynamics remains very challenging. In this paper, we propose to enhance the distinctiveness of the image features by extracting the deep relationship among objects. In particular, we extract objects in the image and construct a deep object relation graph (ORG) to incorporate the semantic connections and relative spatial clues of the objects. We integrate our ORG module into several popular pose regression models. Extensive experiments on various public indoor and outdoor datasets demonstrate that our method improves the performance significantly and outperforms the previous approaches.

show abstract

Section: B Object Related Methodsmentioning

confidence: 99%

Objects Matter: Learning Object Relation Graph for Robust Camera Relocalization

Qiao¹,

Xiang²,

Wang³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Wang et al. [36] further refined the modelling of intra‐ and inter‐modality visual relations in image captioning. Cornia et al.…”

Section: Related Workmentioning

confidence: 99%

Semantic‐meshed and content‐guided transformer for image captioning

Zhang

Sun

et al. 2022

IET Computer Vision

Self Cite

View full text Add to dashboard Cite

The transformer architecture has been the dominant framework for today's image captioning tasks because of its superior performance. However, existing methods based on transformer often lack the integrated use of multi-level semantic information and are weak in maintaining the relevance of captions to the image. In this paper, a semanticmeshed and content-guided transformer network is introduced for image captioning to solve these problems. The semantic-meshed mechanism allows the model to generate words by selecting semantic information of multiple interaction levels adaptively through attention-based reconstruction. And the content-guided module guides the words generation by using attribute features that represent the image content, which aims to keep the generated caption consistent with the main content of the image. Experiments on dataset on the MSCOCO captioning dataset are conducted to validate the authors' model and achieve superior results compared to other state-of-the-art method approaches.

show abstract

“…Huang et al [8], Pan et al [18] rely on variants of attention block to find interactions between multi-modal inputs. Cornia et al [7], Jiayi et al [9], Wang et al [28] use the popular transformer to encode inter-and intra-modal relations by self-attention. Following the success of graphical methods in computer vision community, some researchers adopt scene graph to encode visual relationships with the help of GCNs, which will be talked next.…”

Section: Related Work 21 Image Captioningmentioning

confidence: 99%

“…We compare our models with SCST [21], Up-Down [2], GCN-LSTM [31], VSUA [15], SGAE [30], VRD [22], I 2 RT [28], MMT [7] and GET [9]. Among them, SCST uses reinforcement learning to train the model using its own inference.…”

Section: Quantitative Analysismentioning

confidence: 99%

Improve Image Captioning by Modeling Dynamic Scene Graph Extension

Geng

Zhao

2022

Proceedings of the 2022 International Conference on Multimedia Retrieval

View full text Add to dashboard Cite

Recently, scene graph generation methods have been used in image captioning to encode the objects and their relationships in the encoder-decoder framework, where the decoder selects part of the graph nodes as input for word inference. However, current methods attend to scene graph relying on ambiguous language information, neglecting the strong connections between scene graph nodes. In this paper, we propose a Scene Graph Extension (SGE) architecture to model the dynamic scene graph extension using the partly generated sentence. Our model first uses the generated words and previous attention results of scene graph nodes to make up a partial scene graph. Then we choose objects or relationships that has close connection with the generated graph to infer the next word. Our SGE is appealing in view that it is pluggable to any scene graph based image captioning method. We conduct the extensive experiments on MSCOCO dataset. The results shows that the proposed SGE significantly outperforms the baselines, resulting in a state-of-the-art performance under most metrics. CCS CONCEPTS• Computing methodologies → Image representations; Natural language generation.

show abstract

Improving Intra- and Inter-Modality Visual Relation for Image Captioning

Cited by 18 publications

References 26 publications

Objects Matter: Learning Object Relation Graph for Robust Camera Relocalization

Objects Matter: Learning Object Relation Graph for Robust Camera Relocalization

Semantic‐meshed and content‐guided transformer for image captioning

Improve Image Captioning by Modeling Dynamic Scene Graph Extension

Contact Info

Product

Resources

About