Eating and drinking gesture spotting and recognition using a novel adaptive segmentation technique and a gesture discrepancy measure

Abstract

Despite the increasing developments on human activity recognition using wearable technology, there are still many open challenges in spotting and recognising sporadic gestures. As opposed to activities, which exhibit continuous behaviour, the difficulty of spotting gestures lies in their rather sparse nature. This paper proposes a novel solution to spot and recognise a set of similar eating and drinking gestures from continuous inertial data streams. First, potential segments containing an eating or a drinking gesture are found using a Crossings-based Adaptive Segmentation Technique (CAST). Second, further to the long-established range of features employed in previous human activities recognition research work, a gesture discrepancy measure is proposed to improve the classification performance of the system. At the final step, a range of state-of-the-art classification models is employed for evaluation. Various conclusions can be drawn from the results obtained. First, given the 100% recall achieved at the segmentation step, the CAST can be considered a reliable segmentation technique for spotting drinking and eating gestures which may be employed in future gesture spotting work. Second, the addition of gesture discrepancy as a feature descriptor consistently improves the classification performance of the system. Third, the reliability of the food and drink intake monitoring approach proposed in this work finds support on the out-performance of previous similar

A deep learning based wearable system for food and drink intake recognition

“…A posteriori a Hidden Markov Model (HMM) is used for classification, achieving a classification precision of 73.0% and a classification recall of 79.0%. The work in Anderez et al (2020) proposes an accelerometer-based adaptive segmentation technique (CAST) to identify potential eating and drinking gestures embedded in the continuous sensor readings. A posteriori, a soft Dynamic Time Warping (DTW)based gesture discrepancy measure alongside a hand-crafted feature vector are used to train a range of different classifiers.…”

“…The work in Junker et al (2008) obtains an 80% spotting recall. Besides, accurate eating and drinking recognition systems still rely on specific domain knowledge (Anderez et al 2020).…”

“…The aim of signal segmentation is to either break down the signal into segments that share a common characteristic or to filter out unwanted segments of the signal. In this case, an adaptive segmentation technique, namely the CAST (Anderez et al 2020), is employed to identify potential segments containing an eating or a drinking gesture. This technique makes use of the crossings between two moving averages ȳ 1 [t] and ȳ 2 [t] (fast and slow respectively) to identify those potential eating and drinking gestures.…”

“…The rapid technological development in ubiquitous computing seen in recent years is translating into an increasing research attention towards Human Activity Recognition (HAR) (Lee et al 2011;Gayathri et al 2015;Adama et al 2018;Ortega-Anderez et al 2019;Anderez et al 2020;Casella et al 2020). Current portable or wearable devices such as smart-phones and smart-watches already integrate a broad array of sensors (i.e.…”

“…First, the Crossings-based Adaptive Segmentation Technique (CAST) (Anderez et al 2018b(Anderez et al , 2020 is employed for spotting potential eating and drinking gestures from the continuous accelerometer readings. Despite the sparse occurrence of food and drink intake gestures, this segmentation technique has shown the capability of retrieving 100% of the eating and drinking gestures embedded within accelerometer signals.…”