An Efficient Approach for Geo-Multimedia Cross-Modal Retrieval

Zhu, Lei; Long, Jun; Zhang, Chengyuan; Yu, Weiren; Yuan, Xinpan; Sun, Longzhi

doi:10.1109/access.2019.2940055

Cited by 9 publications

(4 citation statements)

References 87 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…An overview of our method MMACMR is depicted in Figure 2. Following prevailing solutions [29,49], the backbone of MMACMR comprises an image encoder f v (•; θ v ) and a recipe encoder f r (•; θ r ) which project food images and recipes into a common feature subspace. In this subspace, the cross-modal features can be aligned effectively so that the similarity between images and recipes can be measured with accuracy.…”

Section: Framework Overviewmentioning

confidence: 99%

See 1 more Smart Citation

Disambiguity and Alignment: An Effective Multi-Modal Alignment Method for Cross-Modal Recipe Retrieval

Zou,

Zhu,

Zhu

et al. 2024

Foods

Self Cite

View full text Add to dashboard Cite

As a prominent topic in food computing, cross-modal recipe retrieval has garnered substantial attention. However, the semantic alignment across food images and recipes cannot be further enhanced due to the lack of intra-modal alignment in existing solutions. Additionally, a critical issue named food image ambiguity is overlooked, which disrupts the convergence of models. To these ends, we propose a novel Multi-Modal Alignment Method for Cross-Modal Recipe Retrieval (MMACMR). To consider inter-modal and intra-modal alignment together, this method measures the ambiguous food image similarity under the guidance of their corresponding recipes. Additionally, we enhance recipe semantic representation learning by involving a cross-attention module between ingredients and instructions, which is effective in supporting food image similarity measurement. We conduct experiments on the challenging public dataset Recipe1M; as a result, our method outperforms several state-of-the-art methods in commonly used evaluation criteria.

show abstract

Section: Framework Overviewmentioning

confidence: 99%

“…In line with prior research [49], we use food images with a depth of three channels in the RGB color space. All the images in our experiments are resized to 256 pixels in their shorter dimension and then cropped to 224 × 224 pixels.…”

Section: Implementation Detailsmentioning

confidence: 99%

Disambiguity and Alignment: An Effective Multi-Modal Alignment Method for Cross-Modal Recipe Retrieval

Zou,

Zhu,

Zhu

et al. 2024

Foods

Self Cite

View full text Add to dashboard Cite

show abstract

“…Cross-modal retrieval is a significant problem in the area of multimedia computing [19][20][21][22][23][24][25][26], which aims to find out the similar enough objects of one modality in the multimedia database by a query of different modality. Due to the exponential growth of amount of multimedia data, this task attracts a large number of attentions in recent years.…”

Section: Cross-modal Retrievalmentioning

confidence: 99%

Adversarial Learning Based Semantic Correlation Representation for Cross-Modal Retrieval

Zhu¹,

Song²,

Wei³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

With the rapid development of Internet and the widely usage of smart devices, massive multimedia data are generated, collected, stored and shared on the Internet. This trend makes cross-modal retrieval problem become a hot issue in this years. Many existing works pay attentions on correlation learning to generate a common subspace for cross-modal correlation measurement, and others uses adversarial learning technique to abate the heterogeneity of multi-modal data. However, very few works combine correlation learning and adversarial learning to bridge the inter-modal semantic gap and diminish cross-modal heterogeneity. This paper propose a novel cross-modal retrieval method, named ALSCOR, which is an end-to-end framework to integrate cross-modal representation learning, correlation learning and adversarial. CCA model, accompanied by two representation model, VisNet and TxtNet is proposed to capture non-linear correlation. Beside, intra-modal classifier and modality classifier are used to learn intra-modal discrimination and minimize the inter-modal heterogeneity. Comprehensive experiments are conducted on three benchmark datasets. The results demonstrate that the proposed ALSCOR has better performance than the state-of-the-arts.

show abstract

“…The other challenge is inefficient index and retrieval algorithm in massive geo-multimedia database. To overcome this problem, in Reference [41], a novel hybrid index called GMR-Tree is developed that is an extension of R-Tree by integrating cross-modal representation. However, this work ignores the cross-modal hashing representation that can enhance the search efficiency significantly.…”

Section: Introductionmentioning

confidence: 99%

TDCMR: Triplet-Based Deep Cross-Modal Retrieval for Geo-Multimedia Data

et al. 2021

Self Cite

View full text Add to dashboard Cite

Mass multimedia data with geographical information (geo-multimedia) are collected and stored on the Internet due to the wide application of location-based services (LBS). How to find the high-level semantic relationship between geo-multimedia data and construct efficient index is crucial for large-scale geo-multimedia retrieval. To combat this challenge, the paper proposes a deep cross-modal hashing framework for geo-multimedia retrieval, termed as Triplet-based Deep Cross-Modal Retrieval (TDCMR), which utilizes deep neural network and an enhanced triplet constraint to capture high-level semantics. Besides, a novel hybrid index, called TH-Quadtree, is developed by combining cross-modal binary hash codes and quadtree to support high-performance search. Extensive experiments are conducted on three common used benchmarks, and the results show the superior performance of the proposed method.

show abstract

An Efficient Approach for Geo-Multimedia Cross-Modal Retrieval

Cited by 9 publications

References 87 publications

Disambiguity and Alignment: An Effective Multi-Modal Alignment Method for Cross-Modal Recipe Retrieval

Disambiguity and Alignment: An Effective Multi-Modal Alignment Method for Cross-Modal Recipe Retrieval

Adversarial Learning Based Semantic Correlation Representation for Cross-Modal Retrieval

TDCMR: Triplet-Based Deep Cross-Modal Retrieval for Geo-Multimedia Data

Contact Info

Product

Resources

About