A Multitask Sign Language Recognition System Using Commodity Wi-Fi

Abstract

Wi-Fi sensing for gesture recognition systems is a fascinating and challenging research topic. We propose a multitask sign language recognition framework called Wi-SignFi, which accounts for gestures in the real world associated with various objects, actions, or scenes. The proposed framework comprises a convolutional neural network (CNN) and K-nearest neighbor (KNN) module. It is evaluated on the public SignFi dataset and achieves 98.91%, 86.67%, and 99.99% average gesture recognition accuracies on 276/150 activities, five users, and two environments, respectively. The experimental results show that the proposed gesture recognition method outperforms previous methods. Instead of converting the channel state information (CSI) data of multiple antennas into three-dimensional matrices (i.e., color images) as in the existing literature, we found that the CSI data can be converted into matrices (i.e., grayscale images) by concatenating different channels, allowing the Wi-SignFi model to balance between speed and accuracy. This finding facilitates deploying Wi-SignFi on Nvidia’s Jetson Nano edge embedded devices. We expect this work to promote the integration of Wi-Fi sensing and the Internet of Things (IoT) and improve the quality of life of the deaf community.

Revolutionizing Accessibility: Smart Wheelchair Robot and Mobile Application for Mobility, Assistance, and Home Management

“…Zheng et al (2020) [14] propose a transformer-based sign language translation model for Chinese Sign Language with an impressive 92% accuracy, yet a more in-depth exploration of the inherent complexities and limitations of sign language-to-text translation is warranted. Gao et al's (2023) paper [15] introduces a novel approach using Wi-Fi signals for multitask sign language recognition in Chinese Sign Language, but a fuller discussion of the environmental constraints and Wi-Fi signal limitations is needed for realworld implementation. Mannan et al (2022) [16] achieve an exceptional 97% accuracy in Chinese sign language recognition through hyperparameter optimization, yet a deeper dive into the computational demands and training resources required is necessary for a comprehensive assessment of practicality.…”

Optimised Machine Learning-based Translation of Indian Sign Language to Text

“…The model works better for smaller datasets and its performance is optimised by lowering the learning rate. A multitask sign language recognition framework built on CNN and K-nearest neighbour (KNN) module was proposed by [13]. The novel concept of Wireless sensing has been used for sign language recognition, although the model achieve an accuracy of 99.9% on smaller datasets but it took higher time to train the model.…”