Smartphone-based food recognition system using multiple deep CNN models

Abstract

People with blindness or low vision utilize mobile assistive tools for various applications such as object recognition, text recognition, etc. Most of the available applications are focused on recognizing generic objects. And they have not addressed the recognition of food dishes and fruit varieties. In this paper, we propose a smartphone-based system for recognizing the food dishes as well as fruits for children with visual impairments. The Smartphone application utilizes a trained deep CNN model for recognizing the food item from the real-time images. Furthermore, we develop a new deep convolutional neural network (CNN) model for food recognition using the fusion of two CNN architectures. The new deep CNN model is developed using the ensemble learning approach. The deep CNN food recognition model is trained on a customized food recognition dataset.The customized food recognition dataset consists of 29 varieties of food dishes and fruits. Moreover, we analyze the performance of multiple state of art deep CNN models for food recognition using the transfer learning approach. The ensemble model performed better than state of art CNN models and achieved a food recognition accuracy of 95.55 % in the customized food dataset. In addition to that, the proposed deep CNN model is evaluated in two publicly available food datasets to display its efficacy for food recognition tasks.

Accurate Real-Life Chinese Dish Recognition

“…A dozen of works have concerned the deployment of food recognition systems on smartphone or on cloud for realworld dish image analysis [32][33][34][35][36][37][38][39][40][41][42][43][44][45][46][47][48][49]. As to multistage approaches, Kawano and Yanai implemented a food recognition system on smartphone with the purpose of recording calories, nutrition, and eating habits [32].…”

“…Moreover, it is evaluated on images taken by using hand-held cameras, and its top-five accuracy is also promising. Fakhrou also used smartphone for image acquisition of dishes and fruits as the input of an end-to-end recognition system [40]. e system utilizes ensemble learning to fuse two deep neural architectures, and its accuracy achieves 95.55% on twenty-nine categories of customized dataset that benefits children with visual impairments.…”

Eliminate the hardware: Mobile terminals-oriented food recognition and weight estimation system

“…The sample size used for Aliyun Cloud food recognition model verification is not large enough, and the types of food involved are limited, containing only some common foods. Although the accuracy of food identification has reached 89.6%, a larger sample size is needed to further verify and improve the accuracy, and it is expected to achieve all-around and multi-type accurate identification of food ( 7 , 38 ). The area-weight calculation model needs to be further optimized, realizing AR automatic ranging when taking pictures, reducing the requirements for taking pictures, and image processing; further expanding the sample size for training, learning, and verification, and improving more accurate weight estimation.…”

African foods for deep learning-based food recognition systems dataset

“…We developed four deep learning models: Inception-v3, plain convolutional neural network, Mobilenet-v2, and sequential feed-forward neural network. These models were selected based on their successes for western and eastern food recognition systems from related studies [ [19] , [20] , [21] , [22] ]. We evaluated these models based on six significant metrics: training accuracy, validation accuracy, testing accuracy, precision accuracy for individual classes, recall, and f1-score.…”