Food and Ingredient Joint Learning for Fine-Grained Recognition

Liu, Chengxu; Liang, Yuanzhi; Xue, Yao; Qian, Xueming; Fu, Jianlong

doi:10.1109/tcsvt.2020.3020079

Cited by 43 publications

(10 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Many existing methods [19], [20], [21], [22], [23], [24] are based on the development of deep learning and have returned good performances on the aforementioned datasets, working on the dish-level recognition, namely predicting the category of the food image. A few studies [25], [26], [27] introduced the multi-task structures to predict the dish category and corresponding ingredients, however, the location and area of the ingredients cannot be obtained by food image recognition alone.…”

Section: Related Workmentioning

confidence: 99%

HSIFoodIngr-64: A Dataset for Hyperspectral Food-Related Studies and a Benchmark Method on Food Ingredient Retrieval

et al. 2023

View full text Add to dashboard Cite

Food-related issues have attracted increasing attention recently due to its various applications in our daily life, for example, restaurant service and dietary assessment, which are essential for human health management and understanding food characteristics. Many existing datasets are proposed for various food-relevant tasks, e.g., food detection and segmentation. However, most of the datasets are labelled with only dish-level annotations and lack detailed information for the corresponding ingredients. Hyperspectral imaging (HSI), which can explore the emissive characteristics of different objects with a long region of spectral bands, is of great potential for food and ingredient analysis. In the present work, a new food image dataset HSIFoodIngr-64 containing 3,389 pairs of HSI and RGB images with 21 dish classes and 64 ingredient categories was established. It is the first HSI-based food image dataset, of which all images are labelled with dish-level and ingredient-level annotations and corresponding pixel-wise ingredient masks. Therefore, our dataset can be applied to different food-centric tasks. Furthermore, this research proposed a benchmark method on the food ingredient retrieval task, which consists of two sub-networks called IPSN for ingredient prediction sub-network and ERSN for edge refinement sub-network. Extensive experiments were conducted on HSIFoodIngr-64 dataset and the effectiveness of our proposed method was verified. HSIFoodIngr-64 dataset is expected to provide a new perspective on food analysis and inspire more efforts on various food-related issues. We have made the HSIFoodIngr-64 dataset public at: https://doi.org/10.7910/DVN/E7WDNQ.

show abstract

Section: Related Workmentioning

confidence: 99%

HSIFoodIngr-64: A Dataset for Hyperspectral Food-Related Studies and a Benchmark Method on Food Ingredient Retrieval

et al. 2023

View full text Add to dashboard Cite

show abstract

“…Later, Mao et al [9] propose to construct a food hierarchy based on visual similarity and nutrition content [33] without requiring human efforts. In addition, food recognition has been studied under different scenarios such as ingredient recognition [34], [35], fine-grained recognition [36]- [38], fewshot learning [39], [40], long-tailed recognition [14], [41] and continual learning [42], [43]. However, none of the existing method are capable of learning new classes continuously in long-tailed distributions, which closely relates to real-world food recognition as new foods appear sequentially overtime with a minority of foods consumed more frequently than the others [14].…”

Section: A Food Recognitionmentioning

confidence: 99%

Online Continual Learning For Visual Food Classification

Zhu

2021

2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW)

View full text Add to dashboard Cite

Deep learning based food recognition has achieved remarkable progress in predicting food types given an eating occasion image. However, there are two major obstacles that hinder deployment in real world scenario. First, as new foods appear sequentially overtime, a trained model needs to learn the new classes continuously without causing catastrophic forgetting for already learned knowledge of existing food types. Second, the distribution of food images in real life is usually long-tailed as a small number of popular food types are consumed more frequently than others, which can vary in different populations. This requires the food recognition method to learn from classimbalanced data by improving the generalization ability on instance-rare food classes. In this work, we focus on long-tailed continual learning and aim to address both aforementioned challenges. As existing long-tailed food image datasets only consider healthy people population, we introduce two new benchmark food image datasets, VFN-INSULIN and VFN-T2D, which exhibits on the real world food consumption for insulin takers and individuals with type 2 diabetes without taking insulin, respectively. We propose a novel end-to-end framework for long-tailed continual learning, which effectively addresses the catastrophic forgetting by applying an additional predictor for knowledge distillation to avoid misalignment of representation during continual learning. We also introduce a novel data augmentation technique by integrating class-activation-map (CAM) and CutMix, which significantly improves the generalization ability for instancerare food classes to address the class-imbalance issue. The proposed method is evaluated on Food101-LT, VFN-LT, VFN-INSULIN and VFN-T2D and show promising performance with large margin improvements compared with existing methods. We conduct an ablation study and discuss potential techniques that can further improve the performance, demonstrating great potential to deploy our method in real world food recognition applications.

show abstract

“…It is the model's performance on the base class data during the current session, and it is computed by Equation (50).…”

Section: Base Sessionmentioning

confidence: 99%

Progressive Kernel Extreme Learning Machine for Food Image Analysis via Optimal Features from Quality Resilient CNN

Tahir

Loo

2021

Applied Sciences

View full text Add to dashboard Cite

Recently, food recognition has received more research attention for mHealth applications that use automated visual-based methods to assess dietary intake. The goal is to improve the food diaries by addressing the challenges faced by existing methodologies. In addition to the classical challenge of the absence of rigid food structure and intra-class variations, food diaries employing deep networks trained with pristine images are susceptible to quality variations in real-world conditions of image acquisition and transmission. Similarly, existing progressive classifiers that use visual features via a convolutional neural network (CNN) classify food categories and cannot detect food ingredients. We aim to provide a system that selects the optimal subset of features from quality resilient CNNs and subsequently incorporates the parallel type of classification to tackle such challenges. The first progressive classifier recognizes food categories, and its multilabel extension detects food ingredients. Following this idea, after extracting features from the quality resilient category and ingredient CNN models by fine-tuning it on synthetic images generated using the novel online data augmentation method random iterative mixup. Our feature selection strategy uses the Shapley additive explanation (SHAP) values from the gradient explainer to select the best features. Then, novel progressive kernel extreme learning machine (PKELM) is exploited to cater to domain variations due to quality distortions, intra-class variations, and so forth, by remodeling the network structure based on activity value with the nodes. PKELM extension for multilabel classification detects ingredients by employing a bipolar step function to process test output and then selecting the column labels of the resulting matrix with a value of one. Moreover, during online learning, the PKELM novelty detection mechanism can label unlabeled instances and detect noisy samples. Experimental results showed superior performance on an integrated set of measures for seven publicly available food datasets.

show abstract

Food and Ingredient Joint Learning for Fine-Grained Recognition

Cited by 43 publications

References 38 publications

HSIFoodIngr-64: A Dataset for Hyperspectral Food-Related Studies and a Benchmark Method on Food Ingredient Retrieval

HSIFoodIngr-64: A Dataset for Hyperspectral Food-Related Studies and a Benchmark Method on Food Ingredient Retrieval

Online Continual Learning For Visual Food Classification

Progressive Kernel Extreme Learning Machine for Food Image Analysis via Optimal Features from Quality Resilient CNN

Contact Info

Product

Resources

About