End-to-End Dynamic Gesture Recognition Using MmWave Radar

Abstract

Millimeter-wave (mmWave) radar sensors are a promising modality for gesture recognition as they can overcome several limitations of optic sensors typically used for gesture recognition. These limitations include cost, battery consumption, and privacy concerns. This work focuses on finger level (called micro) gesture recognition using mmWave radar. We propose a set of 6 micro-gestures that are not only intuitive and easy to perform for the user but are distinguishable based on Doppler and angle variation in time. For gesture recognition, we propose an end-to-end solution including an activity detection module (ADM) that automatically segments the data and the gesture classifier (GC) that takes the segmented data and predicts the gesture. Both the ADM and GC are based on machine learning (ML) tools. We evaluate the proposed solution using data collected from 11 users and our proposed solution achieves an end-to-end accuracy of 95%. INDEX TERMS Human-computer interface, activity detection, gesture recognition, radar, machine learningConsidering radar characteristics, we select dynamic gestures rather than static hand/finger poses in our solution. Radars have limited resolution in both the angle and range. Due to form-factor constraints, radar modules on mobile devices have only a few antennas, and as a result have limited angle resolution. While large bandwidths are available at mmWave bands, the range resolution is still several centimeters, which is too coarse for differentiating fingers' positions. Fortunately, radars have superb Doppler (speed) measurement capability. The Doppler resolution is inversely proportional to the duration of the radar coherent processing duration, which is a design parameter. The high Doppler resolution enables the radar to capture and potentially distinguish

Analysis of User-Defined Radar-Based Hand Gestures Sensed Through Multiple Materials

“…The average rate falls inside the moderate category (<.3) [76]. There are no referents with a very high rate of agreement, 1 19 =5% has a high agreement, 17 19 =90% has a moderate agreement, and 1 19 =5% receive a low agreement. The referents in this study are not linked to any environment or device, such as a smart home, smartphone, or smart car, therefore participants were not so subject to the legacy bias [77], which also explains the low magnitudes as they were unfamiliar with these functions.…”

“…On one hand, radar-based gesture recognition systems have demonstrated their effectiveness [10], reliability, and robustness [11] despite being influenced by environmental conditions [12]. Most studies [12]- [18] have reported their perfor-mance in terms of high recognition accuracy rates, making dynamic [16], [17], real-time [19]- [21] interaction feasible. High-frequency, wide-bandwidth radars detect even the finest movements [22], such as the fingers of a hand [23], at a reasonable distance [24].…”

Data augmentation and synthesis of radar signatures for ML-based gesture recognition and activity detection from MoCap trajectories

“…Operating between 58 -60 GHz with one transmitter and three receiver antennas, the radar was used to detect and classify macro hand-gestures with a vocabulary size of three: sweep left-to-right, sweep right-toleft, and hand-tap. As is common in frame-based radar processing, we used the Time-Velocity Diagram (TVD) and Time-Angle Diagram (TAD) as described in [4], computed from the radar CIR signal, as input features to an ML classifier to classify the gesture. Additionally, a ML-based activity detection module was used to segment the gestures.…”