I Hear You Eat and Speak: Automatic Recognition of Eating Condition and Food Type, Use-Cases, and Impact on ASR Performance

Abstract

We propose a new recognition task in the area of computational paralinguistics: automatic recognition of eating conditions in speech, i. e., whether people are eating while speaking, and what they are eating. To this end, we introduce the audio-visual iHEARu-EAT database featuring 1.6 k utterances of 30 subjects (mean age: 26.1 years, standard deviation: 2.66 years, gender balanced, German speakers), six types of food (Apple, Nectarine, Banana, Haribo Smurfs, Biscuit, and Crisps), and read as well as spontaneous speech, which is made publicly available for research purposes. We start with demonstrating that for automatic speech recognition (ASR), it pays off to know whether speakers are eating or not. We also propose automatic classification both by brute-forcing of low-level acoustic features as well as higher-level features related to intelligibility, obtained from an Automatic Speech Recogniser. Prediction of the eating condition was performed with a Support Vector Machine (SVM) classifier employed in a leave-one-speaker-out evaluation framework. Results show that the binary prediction of eating condition (i. e., eating or not eating) can be easily solved independently of the speaking condition; the obtained average recalls are all above 90%. Low-level acoustic features provide the best performance on spontaneous speech, which reaches up to 62.3% average recall for multi-way classification of the eating condition, i. e., discriminating the six types of food, as well as not eating. The early fusion of features related to intelligibility with the brute-forced acoustic feature set improves the performance on read speech, reaching a 66.4% average recall for the multi-way classification task. Analysing features and classifier errors leads to a suitable ordinal scale for eating conditions, on which automatic regression can be performed with up to 56.2% determination coefficient.

Using the Bag-of-Audio-Word Feature Representation of ASR DNN Posteriors for Paralinguistic Classification

“…We performed our experiments on the iHEARu-EAT database [27], which contains the utterances of 30 people recorded while speaking during eating. Six types of food were used along with the "no food" class, resulting in seven classes overall.…”

“…For each speaker and food type, seven utterances were recorded; some subjects refused to eat certain types of foods, resulting in a total of 1414 utterances in German. Although this dataset can be used primarily to test machine learning and signal processing techniques, Hantke et al also anticipated several possible future applications [27]. This dataset was later used in the Interspeech ComParE 2015 Eating Condition Sub-Challenge [15].…”

State-of-the-art tasks and achievements of paralinguistic speech analysis systems

“…Degree of Nativeness (DN) Sub-Challenge В аудиофайлах из базы данных iHEARu-EAT [52,60], предоставленной организаторами из Германии, присутствовал один из 6 типов пищи, которую человек ел во время разговора, или она отсутствовала: 1. яблоко (жесткий фрукт), класс «Apple»; 2. банан (мягкий фрукт), класс «Banana»; 3. нектарин (полумягкий фрукт), класс «Nectarine»; 4. бисквит, класс «Biscuit»; 5. чипсы (хрустящая еда), класс «Crisp»; 6. жевательные конфеты, класс «Haribo», а также 7. без пищи во рту, класс «No Food».…”

Relevant Acoustic Group Features for Automatic Sleepiness Recognition

“…Whereas Automatic Speech Recognition (ASR) tries to determine which words are spoken, CP attempts to discover how those words are spoken and thereby gain knowledge about the various aspects and conditions of the speakers, e.g., age, gender, sleepiness, friendliness, etc. Hantke et al, 2016). The Sleepiness Sub-Challenge employs the openSMILE toolkit (Eyben et al, 2010) to generate the 4368 baseline acoustic features from the "Sleepy Language Corpus" (SLC) (Schuller et al, 2011).…”