Recipe1M+: A Dataset for Learning Cross-Modal Embeddings for Cooking Recipes and Food Images

Marín, Javier; Biswas, Aritro; Ofli, Ferda; Hynes, Nicholas; Salvador, Amaia; Aytar, Yusuf; Weber, Ingmar; Torralba, Antonio

doi:10.48550/arxiv.1810.06553

Cited by 9 publications

(22 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The proposed method is trained and evaluated on Recipe1M [22], the largest publicly available multi-modal food database. Recipe1M [17] and with Ingredient Attention based instruction encoding provides over 1 million recipes (ingredients and instructions), accompanied by one or more images per recipe, leading to 13 million images. The large corpus is supplemented with semantic information (1048 meal classes) for injecting an additional source of information in potential models.…”

Section: Materials and Methods 21 Databasementioning

confidence: 99%

See 1 more Smart Citation

Self-Attention and Ingredient-Attention Based Model for Recipe Retrieval from Image Queries

Fontanellaz

Christodoulidis

Mougiakakou

2019

Proceedings of the 5th International Workshop on Multimedia Assisted Dietary Management

View full text Add to dashboard Cite

Direct computer vision based-nutrient content estimation is a demanding task, due to deformation and occlusions of ingredients, as well as high intra-class and low inter-class variability between meal classes. In order to tackle these issues, we propose a system for recipe retrieval from images. The recipe information can subsequently be used to estimate the nutrient content of the meal. In this study, we utilize the multi-modal Recipe1M dataset, which contains over 1 million recipes accompanied by over 13 million images. The proposed model can operate as a first step in an automatic pipeline for the estimation of nutrition content by supporting hints related to ingredient and instruction. Through self-attention, our model can directly process raw recipe text, making the upstream instruction sentence embedding process redundant and thus reducing training time, while providing desirable retrieval results. Furthermore, we propose the use of an ingredient attention mechanism, in order to gain insight into which instructions, parts of instructions or single instruction words are of importance for processing a single ingredient within a certain recipe. Attention-based recipe text encoding contributes to solving the issue of high intra-class/low inter-class variability by focusing on preparation steps specific to the meal. The experimental results demonstrate the potential of such a system for recipe retrieval from images. A comparison with respect to two baseline methods is also presented.

show abstract

Section: Materials and Methods 21 Databasementioning

confidence: 99%

“…The proposed model architecture is based on a multi-path approach for each of the involved input data types namely, instructions, ingredients and images, similarly to [17]. In Figure 2, the overall structure is presented.…”

Section: Model Architecturementioning

confidence: 99%

Self-Attention and Ingredient-Attention Based Model for Recipe Retrieval from Image Queries

Fontanellaz

Christodoulidis

Mougiakakou

2019

Proceedings of the 5th International Workshop on Multimedia Assisted Dietary Management

View full text Add to dashboard Cite

show abstract

“…Recent introductions of large-scale food-related datasets have further accelerated the research improvements on food understanding. Considering the application purpose, the datasets can be categorized into two groups: food recognition [3,38] and cross-modal recipe retrieval [48,37,41,42,7,48]. We focus on recipe retrieval task in this paper, aiming at retrieval relevant cooking recipes with respect to the image query and vise versa.…”

Section: Related Workmentioning

confidence: 99%

“…A typical recipe consists of a list of ingredients and cooking instructions which may not directly align with the appearance of the corresponding food image. Typically, recent efforts have formulated im2recipe as a cross-modal retrieval problem [48,37,6,62], to align matching recipe-image pairs in a shared latent space with retrieval learning approaches. Concretely, prior work builds two independent networks to encode textual recipes (ingredients and cooking instructions) and food images into embeddings respectively.…”

Section: Introductionmentioning

confidence: 99%

MCEN: Bridging Cross-Modal Gap between Cooking Recipes and Dish Images with Latent Variable Model

Liu

et al. 2020

Preprint

View full text Add to dashboard Cite

Nowadays, driven by the increasing concern on diet and health, food computing has attracted enormous attention from both industry and research community. One of the most popular research topics in this domain is Food Retrieval, due to its profound influence on health-oriented applications. In this paper, we focus on the task of cross-modal retrieval between food images and cooking recipes. We present Modality-Consistent Embedding Network (MCEN) that learns modality-invariant representations by projecting images and texts to the same embedding space. To capture the latent alignments between modalities, we incorporate stochastic latent variables to explicitly exploit the interactions between textual and visual features. Importantly, our method learns the cross-modal alignments during training but computes embeddings of different modalities independently at inference time for the sake of efficiency. Extensive experimental results clearly demonstrate that the proposed MCEN outperforms all existing approaches on the benchmark Recipe1M dataset and requires less computational cost.

show abstract

“…Images from many sources are often plated and shot with an artistic intention to increase appeal rather than represent realism. The few large scale and diverse datasets, such as Recipe1M [12], are mined from recipe websites. While these contain valuable dish level, ingredient level, and preparation attribute annotations, they almost always lack annotations for the portion sizes shown in the photos.…”

Section: Introductionmentioning

confidence: 99%

Nutrition5k: Towards Automatic Nutritional Understanding of Generic Food

Thames

Karpur

Norris

et al. 2021

Preprint

View full text Add to dashboard Cite

Understanding the nutritional content of food from visual data is a challenging computer vision problem, with the potential to have a positive and widespread impact on public health. Studies in this area are limited to existing datasets in the field that lack sufficient diversity or labels required for training models with nutritional understanding capability. We introduce Nutrition5k, a novel dataset of 5k diverse, real world food dishes with corresponding video streams, depth images, component weights, and high accuracy nutritional content annotation. We demonstrate the potential of this dataset by training a computer vision algorithm capable of predicting the caloric and macronutrient values of a complex, real world dish at an accuracy that outperforms professional nutritionists. Further we present a baseline for incorporating depth sensor data to improve nutrition predictions. We will publicly release Nutrition5k in the hope that it will accelerate innovation in the space of nutritional understanding.

show abstract

Recipe1M+: A Dataset for Learning Cross-Modal Embeddings for Cooking Recipes and Food Images

Cited by 9 publications

References 0 publications

Self-Attention and Ingredient-Attention Based Model for Recipe Retrieval from Image Queries

Self-Attention and Ingredient-Attention Based Model for Recipe Retrieval from Image Queries

MCEN: Bridging Cross-Modal Gap between Cooking Recipes and Dish Images with Latent Variable Model

Nutrition5k: Towards Automatic Nutritional Understanding of Generic Food

Contact Info

Product

Resources

About