Eat-Radar: Continuous Fine-Grained Intake Gesture Detection Using FMCW Radar and 3D Temporal Convolutional Network With Attention

Abstract

Unhealthy dietary habits are considered as the primary cause of various chronic diseases, including obesity and diabetes. The automatic food intake monitoring system has the potential to improve the quality of life (QoL) of people with dietrelated diseases through dietary assessment. In this work, we propose a novel contactless radar-based approach for food intake monitoring. Specifically, a Frequency Modulated Continuous Wave (FMCW) radar sensor is employed to recognize fine-grained eating and drinking gestures. The fine-grained eating/drinking gesture contains a series of movements from raising the hand to the mouth until putting away the hand from the mouth. A 3D temporal convolutional network with self-attention (3D-TCN-Att) is developed to detect and segment eating and drinking gestures in meal sessions by processing the Range-Doppler Cube (RD Cube). Unlike previous radar-based research, this work collects data in continuous meal sessions (more realistic scenarios). We create a public dataset comprising 70 meal sessions (4,132 eating gestures and 893 drinking gestures) from 70 participants with a total duration of 1,155 minutes. Four eating styles (fork & knife, chopsticks, spoon, hand) are included in this dataset. To validate the performance of the proposed approach, seven-fold cross-validation method is applied. The 3D-TCN-Att model achieves a segmental F1-score of 0.896 and 0.868 for eating and drinking gestures, respectively. The results of the proposed approach indicate the feasibility of using radar for fine-grained eating and drinking gesture detection and segmentation in meal sessions.

RadarTCN: Lightweight Online Classification Network for Automotive Radar Targets Based on TCN

“…As designed in [35], TCN networks are generally built using 1D CNN to extract features of a specific dimension. To directly process 3D data, Chunzhuo Wang et al [38] propose a TCN architecture based on 3D CNN, which directly extracts eating action information from RD cubes and performs classification and recognition of eating actions. Three-dimensional TCN represents a novel attempt, demonstrating TCN's ability to process data in multiple spatial and temporal dimension in parallel.…”

Drinking Gesture Detection Using Wrist-Worn IMU Sensors with Multi-Stage Temporal Convolutional Network in Free-Living Environments

“…Mertes et al [16] developed a strain gauge-based smart plate to detect bites based on the weight change of food. In our recent work [12], a novel FMCW radar-based system was proposed for in-meal bite detection. This system was validated using our public Eat-Radar dataset, which 1.…”

“…It should be noted that part of this dataset was collected together with our previous Eat-Radar project [12] and there is no participant overlap between MO and FD datasets.…”

“…Although such results can indicate the performance of the model, it should be noted that the purpose of bite detection is to count the number of bites, whereas point-wise results are unable to reveal such information. To address this issue, we use a segment-wise evaluation method to evaluate the bite detection, which has been applied in previous study [12]. Fig.…”

“…In this study, we use the term bite to refer to eating gestures and drinking gestures. The definitions of eating gestures and drinking gestures are consistent with the work in [12]. Specifically, they are defined as the action of raising the hand to the mouth with cutlery or a water container until the hand is moved away from the mouth.…”