Cross-model convolutional neural network for multiple modality data representation

Wu, Yongxiang; Wang, Li; Cui, Fan; Zhai, Hongbin; Dong, Bo; Wang, Jim Jing-Yan

doi:10.1007/s00521-016-2824-4

Cited by 6 publications

(3 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…10,14 As a result, the hash algorithm has emerged in the field of cross-modal recovery and has been widely considered by academia and industry for large-scale multimodal data retrieval. [27][28][29][30] Early CMH techniques were employed to acquire cross-modal similarity searches in which a mutual Hamming space is converted to by multimodal information/data and aligning heterogeneous feature spaces. [31][32][33][34][35] Cross-modal similarity sensitive hashing (CMSSH) 37 and cross-view hashing (CVH) 36 are early representatives of this type of method.…”

Section: Cmh Methodsmentioning

confidence: 99%

“…There has been observed a significant increment in mapping cross‐modal information to a mutual subspace because of the sudden development of high‐latitude data 10,14 . As a result, the hash algorithm has emerged in the field of cross‐modal recovery and has been widely considered by academia and industry for large‐scale multimodal data retrieval 27–30 . Early CMH techniques were employed to acquire cross‐modal similarity searches in which a mutual Hamming space is converted to by multimodal information/data and aligning heterogeneous feature spaces 31–35 .…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Cross‐modal retrieval based on deep regularized hashing constraints

Khan

Hayat

Wen

et al. 2022

Int J of Intelligent Sys

View full text Add to dashboard Cite

Cross‐modal retrieval has attracted great attention due to the increasing demand for tremendous amounts of multimodal data in recent years. These retrievals could either be text‐to‐image or image‐to‐text. To address the problem of inappropriate information included between images and texts, we propose two cross‐modal recovery techniques established on a dual‐branch neural network defined on a common subspace and the hashing learning method. First, a cross‐modal recovery technique established on a multilabel information deep ranking model (MIDRM) is provided. In this method, we introduce a triplet‐loss function into the dual‐branch neural network model. This function takes advantage of the semantic information of the bimodal components, focusing on not only the similarities between similar images and text features but also the distances between dissimilar images and texts. Second, we establish a new cross‐modal hashing technique said to be the deep regularized hashing constraint (DRHC). In this method, the regularized function is used to replace the binary constraint, and the discrete value is constrained to a certain numerical range so that the network can achieve end‐to‐end training. Overall, the time complexity is greatly improved, and the occupied storage space is also greatly reduced. Different experiments on our proposed MIDRM and DRHC models demonstrate their superior performance to those of the state‐of‐the‐art methods on two widely used data sets. The experimental results show that our approach also increases the mean average precision of cross‐modal recovery.

show abstract

Section: Cmh Methodsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Cross‐modal retrieval based on deep regularized hashing constraints

Khan

Hayat

Wen

et al. 2022

Int J of Intelligent Sys

View full text Add to dashboard Cite

show abstract

“…Its advantages are unified framework and easy implementation; its disadvantages are that compatibility and accuracy are difficult to choose from. Wu et al [30] proposed a new convolutional neural network data representation method for representing different forms of data. And learn the CNN model of each modal data, map different modal data to a public space, and regularize the new representation in the public space through a cross-model correlation matrix.…”

Section: Image-text Retrieval Model Based On Subspace Learning Methodsmentioning

confidence: 99%

Cross-Model Hashing Retrieval Based on Deep Residual Network

Li¹,

Xu²,

Zhang³

et al. 2021

Computer Systems Science and Engineering

View full text Add to dashboard Cite

In the era of big data rich in We Media, the single mode retrieval system has been unable to meet people's demand for information retrieval. This paper proposes a new solution to the problem of feature extraction and unified mapping of different modes: A Cross-Modal Hashing retrieval algorithm based on Deep Residual Network (CMHR-DRN). The model construction is divided into two stages: The first stage is the feature extraction of different modal data, including the use of Deep Residual Network (DRN) to extract the image features, using the method of combining TF-IDF with the full connection network to extract the text features, and the obtained image and text features used as the input of the second stage. In the second stage, the image and text features are mapped into Hash functions by supervised learning, and the image and text features are mapped to the common binary Hamming space. In the process of mapping, the distance measurement of the original distance measurement and the common feature space are kept unchanged as far as possible to improve the accuracy of Cross-Modal Retrieval. In training the model, adaptive moment estimation (Adam) is used to calculate the adaptive learning rate of each parameter, and the stochastic gradient descent (SGD) is calculated to obtain the minimum loss function. The whole training process is completed on Caffe deep learning framework. Experiments show that the proposed algorithm CMHR-DRN based on Deep Residual Network has better retrieval performance and stronger advantages than other Cross-Modal algorithms CMFH, CMDN and CMSSH.

show abstract

Adaptive pedestrian detection by predicting classifier

Tang

et al. 2017

Neural Comput & Applic

View full text Add to dashboard Cite

Cross-model convolutional neural network for multiple modality data representation

Cited by 6 publications

References 38 publications

Cross‐modal retrieval based on deep regularized hashing constraints

Cross‐modal retrieval based on deep regularized hashing constraints

Cross-Model Hashing Retrieval Based on Deep Residual Network

Adaptive pedestrian detection by predicting classifier

Contact Info

Product

Resources

About