Attention driven multi-modal similarity learning

Abstract

To learn a function for measuring similarity or relevance between objects is an important machine learning task, referred to as similarity learning. Conventional methods are usually insufficient for processing complex patterns, while more sophisticated methods produce results supported by parameters and mathematical operations that are hard to interpret. To improve both model robustness and interpretability, we propose a novel attention driven multi-modal algorithm, which learns a distributed similarity score over different relation modalities and develops an interaction-oriented dynamic attention mechanism to selectively focus on salient patches of objects of interest. Neural networks are used to generate a set of high-level representation vectors for both the entire object and its segmented patches. Multi-view local neighboring structures between objects are encoded in the high-level object representation through an unsupervised pre-training procedure. By initializing the relation embeddings with object cluster centers, each relation modality can be reasonably interpreted as a semantic topic. A layer-wise training scheme based on a mixture of unsupervised and supervised training is proposed to improve generalization. The effectiveness of the proposed method and its superior performance compared against state-of-the-art algorithms are demonstrated through evaluations based on different image retrieval tasks.

An Interpretable Deep Architecture for Similarity Learning Built Upon Hierarchical Concepts

“…Existing works [61] have shown that, nowadays, although deep neural networks are gradually replacing hand-crafted feature extraction, they do not encourage the use of expert knowledge, for instance, provided by existing feature extraction methods, to enhance the learning. To improve this, we have previously proposed an unsupervised multi-view training algorithm in [22], [62]. It pre-trains a CNN to preserve knowledge offered by multiple image feature extraction methods that characterize heterogeneous properties of the image content.…”

“…Therefore, these images are deemed to be connected under the different modalities of "juice", "tree" and "company". To accommodate this phenomenon, various multimodal similarity learning algorithms have been developed, e.g., by using different kernel functions [19], base metrics [20], transformation functions [21] or distributed relation measures across multiple dimensions [22] in order to model such diverse relation modalities. A brief review on this is provided in Section II-B.…”

Self-attention convolutional neural network for improved MR image reconstruction

“…Along the same direction, the self-attention GAN was constructed for the generation of natural images, where the self-attention mechanism was incorporated into both generator and discriminator of a convolutional GAN. Attention was used in other applications, such as similarity learning (Gao, 2018) and hand gesture recognition (Li 2018).…”

ITNLP-ARC at SemEval-2018 Task 12: Argument Reasoning Comprehension with Attention

“…In most recent years, long short-term memory (LSTM) network (Bengio et al, 1994;Hochreiter and Schmidhuber, 1997), BiLSTM (Pennington et al, 2014a) and gated recurrent unit (GRU) (Cho et al, 2014) are widely used to get sentence representative vector, and achieved better result compared with traditional methods. Attention model also known as alignment model pays more attention to two sentences interaction (Zheng et al, 2018;Gao et al, 2018), which is usually applied in information extraction, relation extraction, text summarization and machine translation. In machine translation, the attention model can be focused on one or a few words of input to make the translation more accurate when generating each new word.…”