A Linguistic Perspective on Radar Micro-Doppler Analysis of American Sign Language

Gürbüz, Sevgi Zübeyde; Gürbüz, Ali Cafer; Malaia, Evie; Griffin, Darrin J.; Crawford, Chris; Rahman, Mohammed Mahbubur; Aksu, Ridvan; Kurtoğlu, Emre; Mdrafi, Robiulhossain; Anbuselvam, Ajaymehul; Macks, Trevor; Ozcelik, Engin

doi:10.1109/radar42522.2020.9114818

Cited by 21 publications

(7 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Radar-based ASL recognition to-date has primarily focused on the recognition of snapshots of specific words or phrases. In [35], ten (10) different ASL phrases that would be relevant to emergency response were recognized with an accuracy of 95% using transfer learning from VGG-16 to classify Xband micro-Doppler signatures. In [12], feature-level fusion of RF sensors operating at three different transmit frequencies (10 GHz, 24 GHz, and 77 GHz) were used together with a random forest classifier trained only an measured micro-Doppler signatures from fluent ASL signers yielded a classification accuracy of 72.5% for 20 ASL signs.…”

Section: B Radar-based Asl Recognitionmentioning

confidence: 99%

ASL Trigger Recognition in Mixed Activity/Signing Sequences for RF Sensor-Based User Interfaces

Kurtoğlu¹,

Gürbüz²,

Malaia³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Section: B Radar-based Asl Recognitionmentioning

confidence: 99%

ASL Trigger Recognition in Mixed Activity/Signing Sequences for RF Sensor-Based User Interfaces

Kurtoğlu¹,

Gürbüz²,

Malaia³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

“…Due to the frequency band, modulation bandwidth and modulated waveforms of different radars, data consistence can not be easily guaranteed. Unlike video, radar measurements are not inherently images, but actually form a time-stream of complex I/Q data from which line-of-sight distance and radial velocity may be computed [20]. In this work, the radar echo signals require further processing to become visualized in the modality of micro-Doppler signatures of motion.…”

Section: State-of-the-art and Characteristic Of Radar Sensingmentioning

confidence: 99%

EM-Sign: A Non-Contact Recognition Method Based on 24 GHz Doppler Radar for Continuous Signs and Dialogues

Lan

Zhang

et al. 2020

Electronics

View full text Add to dashboard Cite

We studied continuous sign language recognition using Doppler radar sensors. Four signs in Chinese sign language and American sign language were captured and extracted by complex empirical mode decomposition (CEMD) to obtain spectrograms. Image sharpening was used to enhance the micro-Doppler signatures of the signs. To classify the different signs, we utilized an improved Yolov3-tiny network by replacing the framework with ResNet and fine-tuned the network in advance. This method can remove the epentheses from the training process. Experimental results revealed that the proposed method can surpass the state-of-the-art sign language recognition methods in continuous sign recognition with a precision of 0.924, a recall of 0.993, an F1-measure of 0.957 and a mean average precision (mAP) of 0.99. In addition, dialogue recognition in three daily conversation scenarios was performed and evaluated. The average word error rate (WER) was 0.235, 10% lower than in of other works. Our work provides an alternative form of sign language recognition and a new approach to simplify the training process and achieve a better continuous sign language recognition effect.

show abstract

“…In previous work [7], we have found that machine learning can be used on RF sensor data to distinguish native ASL signing from copysigning by hearing individuals. This is also known as "imitation" signing, to emphasize the linguistic and kinematic differences that are observable in the RF data [9]. It has been reported that it can take learners of sign language at least 3 years to produce signs in a manner that is perceived as fluent by native signers [10].…”

Section: Introductionmentioning

confidence: 99%

Word-Level ASL Recognition and Trigger Sign Detection with RF Sensors

Rahman

Kurtoğlu

Mdrafi

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

Current research in the recognition of American Sign Language (ASL) has focused on perception using video or wearable gloves. However, deaf ASL users have expressed concern about the invasion of privacy with video, as well as the interference with daily activity and restrictions on movement presented by wearable gloves. In contrast, RF sensors can mitigate these issues as it is a non-contact ambient sensor that is effective in the dark and can penetrate clothes, while only recording speed and distance. Thus, this paper investigates RF sensing as an alternative sensing modality for ASL recognition to facilitate interactive devices and smart environments for the deaf and hardof-hearing. In particular, the recognition of up to 20 ASL signs, sequential classification of signing mixed with daily activity, and detection of a trigger sign to initiate human-computer interaction (HCI) via RF sensors is presented. Results yield %91.3 ASL word-level classification accuracy, %92.3 sequential recognition accuracy, 0.93 trigger recognition rate.

show abstract

A Linguistic Perspective on Radar Micro-Doppler Analysis of American Sign Language

Cited by 21 publications

References 8 publications

ASL Trigger Recognition in Mixed Activity/Signing Sequences for RF Sensor-Based User Interfaces

ASL Trigger Recognition in Mixed Activity/Signing Sequences for RF Sensor-Based User Interfaces

EM-Sign: A Non-Contact Recognition Method Based on 24 GHz Doppler Radar for Continuous Signs and Dialogues

Word-Level ASL Recognition and Trigger Sign Detection with RF Sensors

Contact Info

Product

Resources

About