Transformer Decoders with MultiModal Regularization for Cross-Modal Food Retrieval

Disambiguity and Alignment: An Effective Multi-Modal Alignment Method for Cross-Modal Recipe Retrieval

“…Similar to the majority of previous studies [9,29,44], we sample 1 K and 10 K imagerecipe pairs from the test partition and assess the retrieval performance for image-to-recipe and and recipe-to-image tasks using median rank (MedR) and recall rate at top k (R@k). Among these metrics, MedR represents the median index of the retrieved samples for each query, measuring the ability of models to understand the semantic correlation between two modalities and the accuracy of retrieval.…”

“…Crowdsourcing strategy is also used to construct program representations of recipes [41]. Thanks to the flourishing development of visual language pre-training recently, some pioneers [42][43][44][45] have further embedded complex semantic relationship information into common feature subspace by leveraging the pre-trained Contrastive Language-Image Pre-Training model (CLIP).…”

CREAMY: Cross-Modal Recipe Retrieval By Avoiding Matching Imperfectly

“…For recipes, along with the proposed of various attention mechanism, Bert [44]- [46] and Transformer [47]- [50] are utilized to implement stronger textual encoder than sequential models such as skip-thought [26], [51] and LSTM [23]- [25] involved in early works. Furthermore, cross-modal attention [34], [52], [53] mechanism and large vision-language pre-training models [54]- [58] are employed in cross-modal recipe understanding, which further narrow heterogeneous gap via cross-modal interaction. The difficulty of recipe retrieval mainly stems from complex recipe sample including title, ingredients and instructions, other than a simple phrase or a sentence.…”

“…Following prior works [22], [34], [51], we evaluate the retrieval performance (both image-to-recipe task and recipe-toimage task) using median rank (MedR), which is the median index of the retrieved samples for each query, and recall rate at top-k, representing the percentage of queries for which the correct sample index belongs to the top-k retrieved samples.…”

“…Object detection [26] and image reconstruction [27]- [29] techniques are adopted by previous works to implement a strong-sighted model so that the key visual details can be focused. To better understand the complex semantics from recipes, some works [26], [30] devote to find the key terms in texts, while others [22], [31] attempt to explore the hidden consistent information between different components in recipes, or even capture the interaction of two modalities via cross-modal attention [32]- [34] to enhance cross-modal alignment [35]- [38].…”