Push the Limit of Acoustic Gesture Recognition

Wang, Yanwen; Shen, Jiaxing; Zheng, Yuanqing

doi:10.1109/infocom41043.2020.9155402

Cited by 66 publications

(13 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, in order to provide a truly useful assistive communication tool for ASL signers, it is imperative for ASL recognition systems to consider and assess the performance of sentence-level recognition. This part was largely missed in prior studies [34,61,62] reported in the literature.…”

Section: Sentence-levelmentioning

confidence: 95%

“…This technique, however, requires high power (> 50 watts) speakers and a linear microphone array, which is not suitable for wearable and mobile applications. Wang et al [61] designed a smartphone-based gesture recognition system, which can recognize 15 gestures. To the best of our knowledge, SonicASL is the first of its kind to recognize fine-grained, heterogeneous sign language gestures by leveraging acoustic sensing on commodity earphones and ordinary consumer-grade speakers and microphones.…”

Section: Acoustic-based Sensing Applicationsmentioning

confidence: 99%

“…Based on the Doppler effect, sign language gestures, including both hands and arms, will cause phase and frequency changes of the reflected sonic wave. Some existing works [61,74] utilized channel impulse response (CIR) to extract fine-grained features, however, it requires audio signal encoding and decoding scheme, which costs extra computational resource and processing time. As a real-time communication tool, we make the trade-off between the performance and the latency by using spectrograms.…”

Section: Acousticmentioning

confidence: 99%

See 2 more Smart Citations

SonicASL

Jin

Gao

Zhu

et al. 2021

Proc. ACM Interact. Mob. Wearable Ubiquitous Technol.

View full text Add to dashboard Cite

We propose SonicASL, a real-time gesture recognition system that can recognize sign language gestures on the fly, leveraging front-facing microphones and speakers added to commodity earphones worn by someone facing the person making the gestures. In a user study (N=8), we evaluate the recognition performance of various sign language gestures at both the word and sentence levels. Given 42 frequently used individual words and 30 meaningful sentences, SonicASL can achieve an accuracy of 93.8% and 90.6% for word-level and sentence-level recognition, respectively. The proposed system is tested in two real-world scenarios: indoor (apartment, office, and corridor) and outdoor (sidewalk) environments with pedestrians walking nearby. The results show that our system can provide users with an effective gesture recognition tool with high reliability against environmental factors such as ambient noises and nearby pedestrians.

show abstract

Section: Sentence-levelmentioning

confidence: 95%

Section: Acoustic-based Sensing Applicationsmentioning

confidence: 99%

Section: Acousticmentioning

confidence: 99%

See 1 more Smart Citation

SonicASL

Jin

Gao

Zhu

et al. 2021

Proc. ACM Interact. Mob. Wearable Ubiquitous Technol.

View full text Add to dashboard Cite

show abstract

“…That is even less than the wavelength of the acoustic signal that a commercial speaker can send (1.4 for a 24 signal). Another challenge faced by the existing solutions is the frequency-selective fading effect [54]. Since the transceivers are close to the human face, the system is in a severe multipath environment and the arriving signals may add up destructively.…”

Section: Introductionmentioning

confidence: 99%

“…The system measures the channel impulse response (CIR) magnitude and uses a CNN to classify the CIR tensor into hand gestures. Similar to UltraGesture, RobuCIR[54] detects a hand gesture by measuring the CIR. Different from the previous work, RobuCIR measures the phase and magnitude of CIR.…”

mentioning

confidence: 99%

Acoustic-based Upper Facial Action Recognition for Smart Eyewear

Xie

Zhang

2021

Proc. ACM Interact. Mob. Wearable Ubiquitous Technol.

View full text Add to dashboard Cite

Smart eyewear (e.g., AR glasses) is considered to be the next big breakthrough for wearable devices. The interaction of state-of-the-art smart eyewear mostly relies on the touchpad which is obtrusive and not user-friendly. In this work, we propose a novel acoustic-based upper facial action (UFA) recognition system that serves as a hands-free interaction mechanism for smart eyewear. The proposed system is a glass-mounted acoustic sensing system with several pairs of commercial speakers and microphones to sense UFAs. There are two main challenges in designing the system. The first challenge is that the system is in a severe multipath environment and the received signal could have large attenuation due to the frequency-selective fading which will degrade the system's performance. To overcome this challenge, we design an Orthogonal Frequency Division Multiplexing (OFDM)-based channel state information (CSI) estimation scheme that is able to measure the phase changes caused by a facial action while mitigating the frequency-selective fading. The second challenge is that because the skin deformation caused by a facial action is tiny, the received signal has very small variations. Thus, it is hard to derive useful information directly from the received signal. To resolve this challenge, we apply a time-frequency analysis to derive the time-frequency domain signal from the CSI. We show that the derived time-frequency domain signal contains distinct patterns for different UFAs. Furthermore, we design a Convolutional Neural Network (CNN) to extract high-level features from the time-frequency patterns and classify the features into six UFAs, namely, cheek-raiser, brow-raiser, brow-lower, wink, blink and neutral. We evaluate the performance of our system through experiments on data collected from 26 subjects. The experimental result shows that our system can recognize the six UFAs with an average F1-score of 0.92.

show abstract