FORTE: Few Samples for Recognizing Hand Gestures with a Smartphone-attached Radar

Abstract

Radar sensing technologies offer several advantages over other gesture input modalities, such as the ability to reliably sense human movements, a reasonable deployment cost, insensitivity to ambient conditions such as light, temperature, and the ability to preserve anonymity. These advantages come at the price of high processing complexity mainly due to the spatio-temporal variations of gesture articulation performed by different people. Deep learning methods, such as CNN-LSTM and 3D CNN-LSTM, have a high potential to recognize radar-based gestures but usually require hundreds or thousands of labeled training samples and high processing power. Asking a lot of people to acquire a lot of gestures is particularly tedious and tiring to the point of being unrealistic. To overcome these challenges, we propose FORTE, a hand gesture recognition with few samples based on an optimized CNN architecture working on pre-processed raw data. Using a k=5-fold cross-validation, we define and compare three alternative CNNs for recognizing hand gestures acquired in a semi-mobile context of use with a portable radar attached to a smartphone. The best CNN reaches an accuracy of 94.96% with a precision of 95.92% and a recall of 96.03% for a dataset composed of solely 5 participants producing 2 samples for 20 classes covering 1 pointing, 2 pantomimic, 3 iconic, and 14 semaphoric gestures. We suggest some implications for designing radar-based gestures and we discuss the limitations of this approach.

Analysis of User-Defined Radar-Based Hand Gestures Sensed Through Multiple Materials

“…The Google Soli chip [41] is embedded in a smartphone for recognizing various classes of gestures. There are several radar-based hand tracking systems such as Ether-Pose [42], CW-Radar [43], the Magic Carpet [44] and its Gesture-Sensing Radars Project, GestureVLAD [10], Ingenious [15], Forte [45] RadarSense [26], RadarGesture [13]. Radar-based gesture interaction has been pioneered by the Magic Carpet [44], a Doppler radar used for sensing coarse body gestures by signal processing.…”

“…In view of all this work, we can apply a DL algorithm, but we are keen to put a template-based [50] recognizer [51] to the test because it allows us to customize the gestures by modifying templates only, with a reasonable number of templates [12], and without re-training [52], [53]. Furthermore, we are siding with the mobile radars that are widely available on the market by taking the Walabot DIY 2 radar [3], [5], [15], [26], [45], [54], [55].…”

Unobtrusive hand gesture recognition using ultra-wide band radar and deep learning

“…This uncertainty makes it impractical to rely solely on a single sensor. To differentiate gestures more effectively, the recognition system incorporates multiple sensors [8]. By combining data from multiple IR-UWB radar or other sensors, more concise information is added [7], [9].…”