Analysis of facial expressions in response to basic taste stimuli using artificial intelligence to predict perceived hedonic ratings

Abstract

Taste stimuli can induce a variety of physiological reactions depending on the quality and/or hedonics (overall pleasure) of tastants, for which objective methods have long been desired. In this study, we used artificial intelligence (AI) technology to analyze facial expressions with the aim of assessing its utility as an objective method for the evaluation of food and beverage hedonics compared with conventional subjective (perceived) evaluation methods. The face of each participant (10 females; age range, 21–22 years) was photographed using a smartphone camera a few seconds after drinking 10 different solutions containing five basic tastes with different hedonic tones. Each image was then uploaded to an AI application to achieve outcomes for eight emotions (surprise, happiness, fear, neutral, disgust, sadness, anger, and embarrassment), with scores ranging from 0 to 100. For perceived evaluations, each participant also rated the hedonics of each solution from –10 (extremely unpleasant) to +10 (extremely pleasant). Based on these, we then conducted a multiple linear regression analysis to obtain a formula to predict perceived hedonic ratings. The applicability of the formula was examined by combining the emotion scores with another 11 taste solutions obtained from another 12 participants of both genders (age range, 22–59 years). The predicted hedonic ratings showed good correlation and concordance with the perceived ratings. To our knowledge, this is the first study to demonstrate a model that enables the prediction of hedonic ratings based on emotional facial expressions to food and beverage stimuli.

Predicting Perceived Hedonic Ratings through Facial Expressions of Different Drinks

“…8.1; Noldus Information Technology, Wageningen, The Netherlands). The most significant difference between this study and the previous one [ 30 ] is the use of FaceReader, whereas a different AI application was employed in the previous report. FaceReader processes facial expressions frame-by-frame at 30 Hz and classifies them into seven emotions (neutral, happy, sad, angry, surprised, scared, and disgusted) with scores ranging from 0 (no visible emotion) to 1 (emotion fully present).…”

“…The experimental procedure and taste solutions were essentially the same as those of our previous study [ 30 ]. A group of 29 healthy female volunteers (age range, 18–55 years; mean ± SD, 23.1 ± 7.9) participated in an experiment aimed at assessing the effectiveness of AI in analyzing facial expressions and establishing a formula for predicting hedonic ratings.…”

“…Additionally, participants were asked to assess the overall hedonic rating of the stimulus using a scale ranging from −5 (extremely unpleasant) to +5 (extremely pleasant), with 0 indicating a neutral response, prior to commencing the next tasting session. It is worth noting that in our previous paper [ 30 ], we employed a scale ranging from −10 to +10.…”

“…However, to date, no studies have sought to predict explicit hedonic ratings by analyzing implicit facial expressions. To address this gap, our research group embarked on a pioneering investigation into the potential of predicting the deliciousness of food and beverages through facial expression analysis in 2021 [ 30 ]. In this study, using 10 female students (21–22 years old), our approach utilized facial expression analysis as the dependent variable, quantifying expressions of neutrality, happiness, sadness, surprise, fear, disgust, and anger for each of the five basic tastes (sweetness, saltiness, sourness, bitterness, and umami) in various solutions.…”

Towards Foodservice Robotics: A Taxonomy of Actions of Foodservice Workers and a Critical Review of Supportive Technology

“…To conclude, it is worth noting the relevance (especially for the design of emotionally intelligent/responsive FS robots) of the past literature on Machine Learning [257] and Deep Learning [258] [259] strategies of facial expressions recognition. In addition, recent studies (under food emotion research) are of particular interest for FS robotics, for they have analyzed facial expressions (recognized with CV software applications like FaceReader™) and related these with food and beverage taste ratings [260] [261], identified ethnic differences in various facial expressions resulting from a same food tasting [262], and showed that facial expressions reflect the level of food-specific satiety [263]. These studies not only reveal the viability and usefulness of monitoring human facial expressions for robots that must interact with human FS workers/clients (e.g., to receive feedback on food prepared by the robot), but also give specific insights on how robots can interpretate facial expressions resulting from food stimuli.…”