Fast Graph Convolution Network Based Multi-label Image Recognition via Cross-modal Fusion

Abstract

In multi-label image recognition, it has become a popular method to predict those labels that co-occur in an image via modeling the label dependencies. Previous works focus on capturing the correlation between labels, but neglect to effectively fuse the image features and label embeddings, which severely affects the convergence efficiency of the model and inhibits the further precision improvement of multi-label image recognition. To overcome this shortcoming, in this paper, we introduce Multi-modal Factorized Bilinear pooling (MFB) which works as an efficient component to fuse cross-modal embeddings and propose F-GCN, a fast graph convolution network (GCN) based multi-label image recognition model. F-GCN consists of three key modules: (1) an image representation learning module which adopts a convolution neural network (CNN) to learn and generate image representations, (2) a label co-occurrence embedding module which first obtains the label vectors via the word embeddings technique and then adopts GCN to capture label co-occurrence embeddings and (3) an MFB fusion module which efficiently fuses these cross-modal vectors to enable an end-to-end model with a multi-label loss function. We conduct extensive experiments on two multi-label datasets including MS-COCO and VOC2007. Experimental results demonstrate the MFB component efficiently fuses image representations and label co-occurrence embeddings and thus greatly improves the convergence efficiency of the model. In addition, the performance of image recognition has also been promoted compared with the state-of-the-art methods. CCS CONCEPTS • Computing methodologies → Image representations.

Multi-Graph Based Hierarchical Semantic Fusion for Cross-Modal Representation

“…Xu et al [20] utilized GCN on an affinity graph to capture inherent similarity structure of cross-modal data. Wang et al [21] developed a fast GCN model to realized cross-modal fusion. Jiang et al [22] used knowledge-bridge graph to bridge the cross-modal semantic relations.…”

G-CAM: Graph Convolution Network Based Class Activation Mapping for Multi-label Image Recognition

“…ML-GCN [2] and A-GCN [22] use GCN to learn the correlation between objects and achieve good results on multi-label images. F-GCN [39] utilizes a cross-modal component to fuse image features and label embeddings, which speeds up the model convergence and achieve comparable results as ML-GCN and A-GCN. In addition, EmotionGCN [9] utilizes GCN to model the correlation between emotions for emotion distribution learning and NRDH [38] adopts GCN to learn the similarity between images for image retrieval.…”

Graph Convolutional Multi-modal Hashing for Flexible Multimedia Retrieval

“…By learning the structural similarities between training data points, GCN can integrate the relationships into data features. Formally, GCN takes the correlation matrix A as input and produces the node-level output [38]. e forward propagation process in GCN is described as:…”

“…Graph Convolutional Network Hashing (GCNH) [47] is a profound uni-modal hashing method, which performs spectral convolution operations on input data points and affinity graph to generate a binary embedding. Besides, GCN can also be applied to cross-modal hashing models [10,33,38,40]. Since each data pair, which contains two different modality features, is associated with neighboring data pairs, thus the interdependency of the graph structure helps to preserve the intramodality and inter-modality similarity.…”

“…Several researches incorporate GCN into hashing methods for learning structuralsimilarity-preserving hash codes and further boosting hashing retrieval performance. However, the existing GCN-based hashing methods have made progress in uni-modal hashing [47] and crossmodal hashing [10,33,38,40], but not in multi-modal hashing.…”