CREAMY: Cross-Modal Recipe Retrieval By Avoiding Matching Imperfectly

Abstract

State-of-the-art methods for cross-modal recipe retrieval failed to consider an underlying but challenging issue, i.e., matching imperfectly problem hidden in positive image-recipe pairs, which is a culprit causing over-fitting. To make up this defect, two critical questions-how to effectively recognize and filter out mismatching parts during the model training and how to pick out and preserve as much matching information as possible need to be answered. To do so, this article proposes a novel method-Cross-modal Recipe rEtrieval by Avoiding Matching imperfectlY, abbreviated as CREAMY, which involving a new-designed learning strategy called Non-Matching and Partial-Matching (NMPM) to undertake two tasks: (1) no longer forcibly aligning each positive image-recipe pair but rather capturing the complementary information from negative pairs; (2) delicately picking up and aligning the matchable part in each pair. To the best of our knowledge, this attempt is a pioneer to defeat the matching imperfectly issue for cross-modal recipe retrieval task. Empirical analysis conducted on Recipe1M dataset validates the advantages of CREAMY over several state-of-the-arts. The code is available at: https://github.com/users/pouqual/CREAMY.

Disambiguity and Alignment: An Effective Multi-Modal Alignment Method for Cross-Modal Recipe Retrieval

“…In line with prior research [ 49 ], we use food images with a depth of three channels in the RGB color space. All the images in our experiments are resized to 256 pixels in their shorter dimension and then cropped to pixels.…”

“…An overview of our method MMACMR is depicted in Figure 2 . Following prevailing solutions [ 29 , 49 ], the backbone of MMACMR comprises an image encoder and a recipe encoder which project food images and recipes into a common feature subspace. In this subspace, the cross-modal features can be aligned effectively so that the similarity between images and recipes can be measured with accuracy.…”