Cross-Modal Image-Recipe Retrieval via Intra- and Inter-Modality Hybrid Fusion

Abstract

In recent years, the Internet has stimulated the explosion of multimedia data. Food-related cooking videos, images, and recipes promote the rapid development of food computing. Image-recipe retrieval is an important sub-task in the field of cross-modal retrieval, which focuses on the measurement of the association between food image and recipe (title, ingredients, instructions). Although the existing methods have proposed some feasible solutions to achieve the goal of Image-recipe retrieval, there are still the following issues: 1) complex model structure and time-consuming training process. 2) the lack of information interaction within modalities and information integration between images and recipes. To this end, we propose a novel lightweight framework named Intra-and Inter-Modality Hybrid Fusion (IMHF). Our IMHF model abandons a separate deep vision encoder and utilizes the transformer module to unify the visual and text features. In this way, valuable information from images and recipes can be condensed and the direct information interaction between the two modalities can be promoted. Both the intra-and inter-modality fusion can be realized. Extensive experiment results on the large-scale benchmark dataset Recipe1M demonstrate that our model IMHF with a lightweight architecture is superior to the state-of-the-art approaches. CCS CONCEPTS• Information systems → Information retrieval; • Computing methodologies → Visual content-based indexing and retrieval.

CREAMY: Cross-Modal Recipe Retrieval By Avoiding Matching Imperfectly

“…To this end, several well-known CV models and NLP models are employed to generate highquality embeddings from food images and recipe texts so as to achieve cross-modal alignment. For example, deep convolutional neural networks, such as VGG [17], [39], [40] and ResNet [18], [19], [41], [42], are used in several works for visual information embedding. To further focus on essential visual features, Faster-R-CNN is involved [43] to detect food object.…”

“…For recipes, along with the proposed of various attention mechanism, Bert [44]- [46] and Transformer [47]- [50] are utilized to implement stronger textual encoder than sequential models such as skip-thought [26], [51] and LSTM [23]- [25] involved in early works. Furthermore, cross-modal attention [34], [52], [53] mechanism and large vision-language pre-training models [54]- [58] are employed in cross-modal recipe understanding, which further narrow heterogeneous gap via cross-modal interaction. The difficulty of recipe retrieval mainly stems from complex recipe sample including title, ingredients and instructions, other than a simple phrase or a sentence.…”

Cross-Modal Retrieval between Event-Dense Text and Image

“…et al [33] proposes a recipe encoder based on hierarchical Transformers. Besides, there are some works [5,22,23,52] resort to the Transformer or multi-head self-attention mechanism to learn the direct information interaction between the recipe text and food image, but in this way, the learned embeddings have to be on-the-fly generated and can not be indexed offline for retrieval. However, few of these existing methods focus on the event features in the modality-specific embedding learning and joint embedding learning optimization to enhance the cross-modal alignment and boost the event-dense cross-modal retrieval performance.…”

“…Baselines for Comparison. Fifteen baselines are considered: JESR [34], AMSR [4], AdaMine [2], R 2 GAN [53], ACME [41], MCEN [12], SN [52], CHEF [28], IMHF [22], SCAN [42], RDE-GAN [37], X-MRS [14], HF-ICMA [23], JEMA [48] and Pair+rec [33] .…”

“…They focus on designing a complicated joint embedding learning process during the training phase to preserve the cross-modal information (e.g. cross-modal reconstruction) [12,37,41,53] or enhance the inter-modal interaction [5,22,23,52]. However, few current works explore the event-related feature to deal with this event-dense cross-modal recipe retrieval task, neglecting the fact that the discriminative significance of each event element and each event itself in the recipe text is different, and only the event-related visual regions in the food image can contribute to the performance of cross-modal retrieval.…”