Transformer-Based Cross-Modal Recipe Embeddings with Large Batch Training

Yang, Jing; Chen, Junwen; Yanai, Keiji

doi:10.1007/978-3-031-27818-1_39

Cited by 5 publications

(1 citation statement)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, Refs. [ 34 , 35 , 36 , 37 , 38 ] employed various Generative Adversarial Networks (GANs) to reconstruct information from food images and recipes to bridge the heterogeneity gap across modalities, while refs. [ 39 , 40 ] leveraged multilingual translation to enrich the recipe information.…”

Section: Introductionmentioning

confidence: 99%

Disambiguity and Alignment: An Effective Multi-Modal Alignment Method for Cross-Modal Recipe Retrieval

Zou,

Zhu,

Zhu

et al. 2024

Foods

View full text Add to dashboard Cite

As a prominent topic in food computing, cross-modal recipe retrieval has garnered substantial attention. However, the semantic alignment across food images and recipes cannot be further enhanced due to the lack of intra-modal alignment in existing solutions. Additionally, a critical issue named food image ambiguity is overlooked, which disrupts the convergence of models. To these ends, we propose a novel Multi-Modal Alignment Method for Cross-Modal Recipe Retrieval (MMACMR). To consider inter-modal and intra-modal alignment together, this method measures the ambiguous food image similarity under the guidance of their corresponding recipes. Additionally, we enhance recipe semantic representation learning by involving a cross-attention module between ingredients and instructions, which is effective in supporting food image similarity measurement. We conduct experiments on the challenging public dataset Recipe1M; as a result, our method outperforms several state-of-the-art methods in commonly used evaluation criteria.

show abstract

Section: Introductionmentioning

confidence: 99%

Disambiguity and Alignment: An Effective Multi-Modal Alignment Method for Cross-Modal Recipe Retrieval

Zou,

Zhu,

Zhu

et al. 2024

Foods

View full text Add to dashboard Cite

show abstract

Cross-modal Recipe Retrieval with Hierarchical Transformers and Pretrained Food Image Encoder

Qin,

Zhang,

Song

2024

Lecture Notes in Computer Science

View full text Add to dashboard Cite

CREAMY: Cross-Modal Recipe Retrieval By Avoiding Matching Imperfectly

Zou,

Zhu,

Zhu

et al. 2024

IEEE Access

View full text Add to dashboard Cite

State-of-the-art methods for cross-modal recipe retrieval failed to consider an underlying but challenging issue, i.e., matching imperfectly problem hidden in positive image-recipe pairs, which is a culprit causing over-fitting. To make up this defect, two critical questions-how to effectively recognize and filter out mismatching parts during the model training and how to pick out and preserve as much matching information as possible need to be answered. To do so, this article proposes a novel method-Cross-modal Recipe rEtrieval by Avoiding Matching imperfectlY, abbreviated as CREAMY, which involving a new-designed learning strategy called Non-Matching and Partial-Matching (NMPM) to undertake two tasks: (1) no longer forcibly aligning each positive image-recipe pair but rather capturing the complementary information from negative pairs; (2) delicately picking up and aligning the matchable part in each pair. To the best of our knowledge, this attempt is a pioneer to defeat the matching imperfectly issue for cross-modal recipe retrieval task. Empirical analysis conducted on Recipe1M dataset validates the advantages of CREAMY over several state-of-the-arts. The code is available at: https://github.com/users/pouqual/CREAMY.

show abstract

Transformer-Based Cross-Modal Recipe Embeddings with Large Batch Training

Cited by 5 publications

References 17 publications

Disambiguity and Alignment: An Effective Multi-Modal Alignment Method for Cross-Modal Recipe Retrieval

Disambiguity and Alignment: An Effective Multi-Modal Alignment Method for Cross-Modal Recipe Retrieval

Cross-modal Recipe Retrieval with Hierarchical Transformers and Pretrained Food Image Encoder

CREAMY: Cross-Modal Recipe Retrieval By Avoiding Matching Imperfectly

Contact Info

Product

Resources

About