Cross-Modality Sub-Image Retrieval using Contrastive Multimodal Image Representations

Abstract

Image Representations for Registration (CoMIRs) can be used to reduce the difficult task of image retrieval across modalities to a simpler one of image retrieval within the same modality.• We combine state-of-the-art contrastive representation learning with fast and robust feature extractors such as SURF to create a bag-of-words model for multimodal image retrieval.• Imposing rotation equivariance to the learned joint representations provides high performance of cross-modality retrieval.• The proposed approach outperforms bag-of-words built upon GAN-based image-to-image translations, as well as a recent reverse image search toolkit using shape descriptors, on a publicly available multimodal dataset of brightfield and second harmonic generation microscopy images.