Multimodal Deep Network Embedding With Integrated Structure and Attribute Information

Abstract

Network embedding is the process of learning lowdimensional representations for nodes in a network, while preserving node features. Existing studies only leverage network structure information and focus on preserving structural features. However, nodes in real-world networks often have a rich set of attributes providing extra semantic information. It has been demonstrated that both structural and attribute features are important for network analysis tasks. To preserve both features, we investigate the problem of integrating structure and attribute information to perform network embedding and propose a Multimodal Deep Network Embedding (MDNE) method. MDNE captures the non-linear network structures and the complex interactions among structures and attributes, using a deep model consisting of multiple layers of non-linear functions. Since structures and attributes are two different types of information, a multimodal learning method is adopted to pre-process them and help the model to better capture the correlations between node structure and attribute information. We employ both structural proximity and attribute proximity in the loss function to preserve the respective features and the representations are obtained by minimizing the loss function. Results of extensive experiments on four real-world datasets show that the proposed method performs significantly better than baselines on a variety of tasks, which demonstrate the effectiveness and generality of our method.

Assessment of the Clusterability of Data Using a Multimodal Convolutional Neural Network

“…shapes and edges) through the application of convolutions on successive layers of multiple spatial scales [17]. Multimodal NNs can utilize multiple types of inputs simultaneously [22].…”

“…We require our model to utilize two types of inputs: density grids (Section III-A) and CC vectors (Section III-B). One such architecture that is able to combine different types of representations is called a multimodal neural network (MNN) [22]. For example, whereas pixels and sound waves are difficult to relate to each other, MNN is able to combine audio and video data in a single neural network [18].…”

“…The MCNN is trained using a backpropagation algorithm [22]. To reduce the chance of overfitting, we use the L 2 regularization on the FC layers [4].…”

Multiscale Spatio-Temporal Graph Neural Networks for 3D Skeleton-Based Motion Prediction

“…With hierarchical architectures and end-to-end training fashion, GNNs mainly fall into two perspectives: a spectral perspective and a vertex perspective. From a spectral perspective, graphs are converted into its spectrum signal processing [43]. For example, Spectral CNN [44] used the eigen-decomposition of graph Laplacian.…”

An Effective Dense Co-Attention Networks for Visual Question Answering

“…Visual Question Answering (VQA) is a multimodal research task that aims to answer questions related to the given image. Compared with other multimodal learning tasks (e.g., visual description [ 1 ], visual grounding [ 2 , 3 , 4 ], multimodal embedding learning [ 5 , 6 , 7 , 8 ]), VQA requires a fine-grained semantic understanding of both visual and textual content to predict the correct natural language answer. Therefore, VQA has recently emerged as an extremely challenging task and drawn considerable attention from researchers.…”