American Sign Language Recognition Using RF Sensing

Abstract

Many technologies for human-computer interaction have been designed for hearing individuals and depend upon vocalized speech, precluding users of American Sign Language (ASL) in the Deaf community from benefiting from these advancements. While great strides have been made in ASL recognition with video or wearable gloves, the use of video in homes has raised privacy concerns, while wearable gloves severely restrict movement and infringe on daily life. Methods: This paper proposes the use of RF sensors for HCI applications serving the Deaf community. A multi-frequency RF sensor network is used to acquire non-invasive, non-contact measurements of ASL signing irrespective of lighting conditions. The unique patterns of motion present in the RF data due to the micro-Doppler effect are revealed using time-frequency analysis with the Short-Time Fourier Transform. Linguistic properties of RF ASL data are investigated using machine learning (ML). Results: The information content, measured by fractal complexity, of ASL signing is shown to be greater than that of other upper body activities encountered in daily living. This can be used to differentiate daily activities from signing, while features from RF data show that imitation signing by non-signers is 99% differentiable from native ASL signing. Feature-level fusion of RF sensor network data is used to achieve 72.5% accuracy in classification of 20 native ASL signs. Implications: RF sensing can be used to study dynamic linguistic properties of ASL and design Deaf-centric smart environments for non-invasive, remote recognition of ASL. ML algorithms should be benchmarked on native, not imitation, ASL data.

Open Radar Initiative: Large Scale Dataset for Benchmarking of micro-Doppler Recognition Algorithms

“…Note that high-pass filtering for ground clutter removal was only applied on the 25 GHz FMCW and 7-10 GHz UWB sensors, and not the 77 GHz FMCW sensor. Due to the finer detail offered the higher millimeter wave transmit frequency, we have found that many deep neural networks (DNNs) actually perform better when the ground clutter not filtered out [25]. This is because the filtering operation removes not just the clutter, but also low-frequency signal components, which may be of interest in gesture recognition and other finescale problems, such as incurred in assisted living applications.…”

Word-Level ASL Recognition and Trigger Sign Detection with RF Sensors

“…Compilation of large datasets for training state-of-the-art deep neural networks is difficult when human subjects are involved, not only because of the time involved in measuring numerous iterations of each class, but also because it can be difficult to recruit participants, especially if from a minority population, such as the Deaf community. In previous work [7], 20 native ASL signs were classified with an accuracy of 72.5% using minimum-redundancy maximum-relevance (mRMR) selection of 150 handcrafted features extracted from a five node multi-frequency RF sensor network and a random forest classifier. To surpass this performance with just a single sensor, recent advances in deep learning, which have yielded great advances in related fields [15], can be applied.…”

“…Unfortunately, this approach is not effective because imitation signing does have kinematic flaws that render them distinguishable from native ASL signing. This is evidenced not only by their differentiability using a support vector machine classifier [7], but also by the poor classification approach attained when deep learning is applied. When a convolutional neural network (CNN) is pre-trained on imitation data and fine tuned with 80% of the native ASL-R dataset, only 46.15% accuracy was attained when testing on the remaining 20% of native ASL-R data.…”

“…Equally significant to these sensing limitations is the perspectives the Deaf community towards these sensing technologies. In a focus group we conducted with 7 Deaf participants [7], we found that deaf ASL users felt frustrated by wearable gloves, which they described as inaccurate and invasive, while concerned with video-based surveillance in the home. They also expressed excitement towards the potential to one day have Deaf-friendly personal assistants or non-invasive technology-augmented smart environments.…”

“…Not only can they provide a sensing capability when video or wearables are not effective, but they can also provide direct measurement of signing kinematics via the micro-Doppler (mD) signature [8] and range-Doppler maps. In previous work [7], we have found that machine learning can be used on RF sensor data to distinguish native ASL signing from copysigning by hearing individuals. This is also known as "imitation" signing, to emphasize the linguistic and kinematic differences that are observable in the RF data [9].…”