Image Retrieval and Classification Using Local Distance Functions

Frome, Andrea; Singer, Yoram; Malik, Jitendra

doi:10.7551/mitpress/7503.003.0057

Cited by 77 publications

(7 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In early cross-modal retrieval, employing Canonical Correlation Analysis (CCA) [5] to build a shared space was the primary method to address cross-modal alignment, aiming to maximize the co-occurrence correlation between different modal features [6]- [10]. Following that, researchers proposed a series of methods based on contrast [11], triplet [12], and ranking [13] metric losses, aiming to better learn the cross-modal common feature space. Building on this, VSE++ [14] utilizes hard sample mining, PCME [15] transforms deterministic features into probabilistic ones, and DiVE [16] introduces discrete prompts to enhance the extraction of modal features, further strengthening the alignment capability and robustness of the common space.…”

Section: A Cross-modal Retrievalmentioning

confidence: 99%

Haptic Signal Reconstruction for Cross-Modal Communications

Wei

Shi

Zhou

2022

IEEE Trans. Multimedia

View full text Add to dashboard Cite

Cross-modal medical image-report retrieval task plays a significant role in clinical diagnosis and various medical generative tasks. Eliminating heterogeneity between different modalities to enhance semantic consistency is the key challenge of this task. The current Vision-Language Pretraining (VLP) models, with cross-modal contrastive learning and masked reconstruction as joint training tasks, can effectively enhance the performance of cross-modal retrieval. This framework typically employs dual-stream inputs, using unmasked data for cross-modal contrastive learning and masked data for reconstruction. However, due to task competition and information interference caused by significant differences between the inputs of the two proxy tasks, the effectiveness of representation learning for intra-modal and cross-modal features is limited. In this paper, we propose an efficient VLP framework named Masked Contrastive and Reconstruction (MCR), which takes masked data as the sole input for both tasks. This enhances task connections, reducing information interference and competition between them, while also substantially decreasing the required GPU memory and training time. Moreover, we introduce a new modality alignment strategy named Mapping before Aggregation (MbA). Unlike previous methods, MbA maps different modalities to a common feature space before conducting local feature aggregation, thereby reducing the loss of fine-grained semantic information necessary for improved modality alignment. Qualitative and quantitative experiments conducted on the MIMIC-CXR dataset validate the effectiveness of our approach, demonstrating state-of-the-art performance in medical cross-modal retrieval tasks.

show abstract

Section: A Cross-modal Retrievalmentioning

confidence: 99%

Haptic Signal Reconstruction for Cross-Modal Communications

Wei

Shi

Zhou

2022

IEEE Trans. Multimedia

View full text Add to dashboard Cite

show abstract

“…We expect the distances of these vectors to each other to become increasingly meaningful in the context of our overarching prediction task as we train the actual prediction model f NN (refs. [31][32][33][34][35] ). As our encoders are modules of our prediction model, they are automatically trained each time we apply backpropagation on f NN through stochastic gradient descent.…”

Section: Embedded Feature Vectorsmentioning

confidence: 99%

Enhanced spatio-temporal electric load forecasts using less data with active deep learning

2022

View full text Add to dashboard Cite

“…Before the invention of class activation mapping, researchers proposed to reduce patch distance (Zuo et al, 2014;Frome et al, 2006) to improve classification results. Forme et al (Frome et al, 2006) NNs face both underfitting and overfitting issues based on the size of the dataset and the NN structure. In the proposed method, we can adjust the background data.…”

Section: Improvement With Background Classmentioning

confidence: 99%

Uncertainty Quantification Neural Network from Similarity and Sensitivity

Kabir

Khosravi

Nahavandi

et al. 2020

2020 International Joint Conference on Neural Networks (IJCNN)

View full text Add to dashboard Cite

Multitask learning is a popular approach to training high-performing neural networks with improved generalization. In this paper, we propose a background class to achieve improved generalization at a lower computation compared to multitask learning to help researchers and organizations with limited computation power. We also present a methodology for selecting background images and discuss potential future improvements. We apply our approach to several datasets and achieved improved generalization with much lower computation. We also investigate class activation mappings (CAMs) of the trained model and observed the tendency towards looking at a bigger picture in a few class classification problems with the proposed model training methodology. Applying transformer with the proposed background class, we receive state-ofthe-art (SOTA) performance on STL-10, CIFAR-10, CIFAR-100, Oxford-102, Caltech-101, and CINIC-10 datasets. Example scripts are available in the 'CAM' folder of the following GitHub Repository: github.com/dipuk0506/UQ

show abstract

Image Retrieval and Classification Using Local Distance Functions

Cited by 77 publications

References 17 publications

Haptic Signal Reconstruction for Cross-Modal Communications

Haptic Signal Reconstruction for Cross-Modal Communications

Enhanced spatio-temporal electric load forecasts using less data with active deep learning

Uncertainty Quantification Neural Network from Similarity and Sensitivity

Contact Info

Product

Resources

About