A deep learning based wearable system for food and drink intake recognition

Abstract

Eating difficulties and the subsequent need for eating assistance are a prevalent issue within the elderly population. Besides, a poor diet is considered a confounding factor for developing chronic diseases and functional limitations. Driven by the above issues, this paper proposes a wrist-worn tri-axial accelerometer based food and drink intake recognition system. First, an adaptive segmentation technique is employed to identify potential eating and drinking gestures from the continuous accelerometer readings. A posteriori, a study upon the use of Convolutional Neural Networks for the recognition of eating and drinking gestures is carried out. This includes the employment of three time series to image encoding frameworks, namely the signal spectrogram, the Markov Transition Field and the Gramian Angular Field, as well as the development of various multi-input multi-domain networks. The recognition of the gestures is then tackled as a 3-class classification problem (‘Eat’, ‘Drink’ and ‘Null’), where the ‘Null’ class is composed of all the irrelevant gestures included in the post-segmentation gesture set. An average per-class classification accuracy of 97.10% was achieved by the proposed system. When compared to similar work, such accurate classification performance signifies a great contribution to the field of assisted living.

Enhancing Nutrition Care Through Real-Time, Sensor-Based Capture of Eating Occasions: A Scoping Review

“…Fifty-three unique devices and four combinations of devices ( 77 – 80 ) identified from 54 studies were included in the data extraction table ( Supplementary Table 1 ). Of the 53 unique devices, 18 devices were placed on the wrist ( 27 – 44 ); nine were worn around the neck ( 45 – 52 ); nine were placed in or around the ears ( 53 – 61 ); seven were glasses-like devices ( 62 – 68 ); and ten were categorized as “Other Devices” ( 30 , 36 , 69 – 76 ) where devices were worn in less common locations of the body or were non-wearable ( 69 , 72 ).…”

“…Therefore, a major advantage of wrist devices is the potential for it to detect both eating and drinking as it does not rely on chewing or swallowing activity. Both food and beverage intake were able to be identified in 5/18 wrist devices ( 27 , 33 , 37 , 40 , 44 ) and three of these could also distinguish between eating and drinking movements ( 27 , 33 , 40 ). The algorithms used by the wrist devices to detect HTM movements could be adapted to different handedness depending on the individual.…”

“…The main disadvantages of the studies that used wrist devices included: generating false positives for hand actions close to the face ( 33 , 42 ), and using a limited number of foods, utensils, and/or behaviors to train and test the model, leading to overly optimistic results ( 27 , 30 , 34 , 36 , 38 ); and users being required to wear the wrist device on both wrists ( 35 ).…”

“…This may be because beverages that are consumed continuously throughout the day such as water, tea, and coffee are difficult to detect ( 67 ) and distinguish from eating using current sensors ( 62 , 80 ). Most of the devices that were able to distinguish between eating and drinking were tested on a limited number of foods and beverages ( 27 , 37 , 40 , 44 , 47 , 49 , 50 , 76 , 78 ), using differences in chewing and swallowing patterns ( 47 , 49 , 50 , 78 ), sounds ( 49 , 76 ), or wrist movements ( 27 , 37 , 44 ) to distinguish between the two. Additional evaluation using more food types in settings where subjects are free to choose their own foods and activities is required.…”

An Analysis of Fluid Intake Assessment Approaches for Fluid Intake Monitoring System

“…In wrist-worn-based approaches, individuals wear sensors on their right or left wrists to collect activity data. Detection algorithms are then applied to recognize drinking activities from the collected data [2,15,16]. In smart-container-based approaches, sensors are either attached to or embedded within a container to record data.…”