Food intake monitoring: an acoustical approach to automated food intake activity detection and classification of consumed food

Abstract

Obesity and nutrition-related diseases are currently growing challenges for medicine. A precise and timesaving method for food intake monitoring is needed. For this purpose, an approach based on the classification of sounds produced during food intake is presented. Sounds are recorded non-invasively by miniature microphones in the outer ear canal. A database of 51 participants eating seven types of food and consuming one drink has been developed for algorithm development and model training. The database is labeled manually using a protocol with introductions for annotation. The annotation procedure is evaluated using Cohen's kappa coefficient. The food intake activity is detected by the comparison of the signal energy of in-ear sounds to environmental sounds recorded by a reference microphone. Hidden Markov models are used for the recognition of single chew or swallowing events. Intake cycles are modeled as event sequences in finite-state grammars. Classification of consumed food is realized by a finite-state grammar decoder based on the Viterbi algorithm. We achieved a detection accuracy of 83% and a food classification accuracy of 79% on a test set of 10% of all records. Our approach faces the need of monitoring the time and occurrence of eating. With differentiation of consumed food, a first step toward the goal of meal weight estimation is taken.

Wearable Food Intake Monitoring Technologies: A Comprehensive Review

“…Pabler and Fischer [82] further utilized this method by developing a low computational cost algorithm for detecting food intake sound. Similarly, Pabler et al [83] took advantage of chewing sound in the breakdown process detected by the sensor system in the outer ear canal. Other studies that are likewise capable of detecting chewing events with the count numbers of chewing and food texture classification are proposed in [84,85].…”

Mobile Ecological Momentary Diet Assessment Methods for Behavioral Research: Systematic Review (Preprint)

“…Because of the known limitation of existing dietary assessment methods, the research community is motivated to develop new solutions aimed at (semi-)automating the assessment of dietary intake. While the automated methods of real-time image-based detection [14][15][16][17][18][19][20][21][22][23][24] and real-time detection of food intake by biomechanical sensors or hand-held devices [25][26][27][28][29][30][31][32][33][34][35][36][37][38][39][40][41][42] have seen significant progress [15,24] in terms of identifying foods and estimating portion sizes [14][15][16][17][18][19][20][21][22][23][24] detecting wrist or hand motion [25][26][27][28]…”

“…image-assisted and image-based assessment [14][15][16][17][18][19][20][21][22][23][24] and the detection of food intake by biomechanical sensors or hand-held devices [25][26][27][28][29][30][31][32][33][34][35][36][37][38][39][40][41][42]. Significant progress has been made in image-assisted and image-based food recording that has resulted in the improved accuracy of dietary self-report [15,24].…”

“…Significant progress has been made in image-assisted and image-based food recording that has resulted in the improved accuracy of dietary self-report [15,24]. Similarly, wearable technology including gyroscopes, microphones, and mechanical or electrical impedance sensors have been adapted to detect wrist or hand motion [25][26][27][28][29][30][31] or patterns of chewing or swallowing indicative of food intake (eg, number of bites) [32][33][34][35][36][37][38][39][40][41]. However, design and proof-of-concept data suggest the automation of dietary assessment remains out of reach for the time being.…”

“…However, design and proof-of-concept data suggest the automation of dietary assessment remains out of reach for the time being. For instance, the mean detection accuracy of image detection and wearable devices have been acceptable in controlled settings (range = 73-99%) [18,26,30,31,34,35,[38][39][40][41][42][43][44][45][46][47][48] but limited testing has been done in natural settings [27].…”