Attention-Augmented Electromagnetic Representation of Sign Language for Human-Computer Interaction in Deaf-and-Mute Community

Lan, Shengchang; Ye, Linting; Zhang, Kang

doi:10.23919/usnc-ursi51813.2021.9703456

Cited by 3 publications

(3 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The application works in offline mode and uses ASL and Filipino Sign Language (FPL) datasets. A PC-based prototype is presented by authors in [28] for detecting hand images using sensors. An android-based application is presented in [29], where visual feedback is provided to the user.…”

Section: System Level Reviewmentioning

confidence: 99%

“…Hardware/Software Summary [2] LMD, Android app D-M to ND-M to D-M, Single SLD [3] PC, Harris algorithm D-M to ND-M to D-M, Single SLD [4] Raspberry Pi D-M to ND-M, Single SLD [5] LMD, LSTM Neural networks D-M to ND-M, Single SLD [6] LMD, (GMM, HMM, DTW) D-M to ND-M, Single SLD [7] PC, NN algorithms D-M to ND-M, Single SLD [8] PC, Arduino, Gloves, Flex sensors Calibration required for the custom-made product, D-M to ND-M, Single SLD [10] Arduino, Gloves, Sensors Calibration required for the custom-made product, D-M to ND-M, Single SLD [12] PC, TensorFlow D-M to ND-M, Single SLD [13] PC, Various ML algorithms D-M to ND-M to D-M, Single SLD [14] PC, TensorFlow, CNN D-M to ND-M, Single SLD [15] Android D-M to ND-M, Multi SLD [16] Android D-M to ND-M, Single SLD [17] PC, CNN D-M to ND-M, Single SLD [18] Arduino, Gloves, Flex sensors Calibration required for the custom-made product, D-M to ND-M, Single SLD [19] PC NN D-M to ND-M, Single SLD [20] Gloves, Flex sensors, Voice recorder, Matlab D-M to ND-M, Single SLD [21] PC, CNN D-M to ND-M, Single SLD [22] CNN D-M to ND-M to D-M, Single SLD [23] Gloves, Sensors, Arduino, KNN D-M to ND-M, Single SLD [24] PC-based, PCA D-M to ND-M, Single SLD [25] Arduino, Gloves, Sensors, KNN Calibration required for the custom-made product, D-M to ND-M, Single SLD [26] PC-based, CNN D-M to ND-M, Single SLD [27] Android application D-M to ND-M, Multi SLD [28] PC, CNN D-M to ND-M, Single SLD [29] Android D-M to ND-M to D-M, Single SLD…”

Section: Referencesmentioning

confidence: 99%

See 1 more Smart Citation

A Machine Learning Based Full Duplex System Supporting Multiple Sign Languages for the Deaf and Mute

et al. 2023

View full text Add to dashboard Cite

This manuscript presents a full duplex communication system for the Deaf and Mute (D-M) based on Machine Learning (ML). These individuals, who generally communicate through sign language, are an integral part of our society, and their contribution is vital. They face communication difficulties mainly because others, who generally do not know sign language, are unable to communicate with them. The work presents a solution to this problem through a system enabling the non-deaf and mute (ND-M) to communicate with the D-M individuals without the need to learn sign language. The system is low-cost, reliable, easy to use, and based on a commercial-off-the-shelf (COTS) Leap Motion Device (LMD). The hand gesture data of D-M individuals is acquired using an LMD device and processed using a Convolutional Neural Network (CNN) algorithm. A supervised ML algorithm completes the processing and converts the hand gesture data into speech. A new dataset for the ML-based algorithm is created and presented in this manuscript. This dataset includes three sign language datasets, i.e., American Sign Language (ASL), Pakistani Sign Language (PSL), and Spanish Sign Language (SSL). The proposed system automatically detects the sign language and converts it into an audio message for the ND-M. Similarities between the three sign languages are also explored, and further research can be carried out in order to help create more datasets, which can be a combination of multiple sign languages. The ND-M can communicate by recording their speech, which is then converted into text and hand gesture images. The system can be upgraded in the future to support more sign language datasets. The system also provides a training mode that can help D-M individuals improve their hand gestures and also understand how accurately the system is detecting these gestures. The proposed system has been validated through a series of experiments resulting in hand gesture detection accuracy exceeding 95%.

show abstract

Section: System Level Reviewmentioning

confidence: 99%

Section: Referencesmentioning

confidence: 99%

A Machine Learning Based Full Duplex System Supporting Multiple Sign Languages for the Deaf and Mute

et al. 2023

View full text Add to dashboard Cite

show abstract

“…These cues provide important information about our emotions [5] and convey information which words alone cannot express [20]. The automatic analysis of body language is widely studied in the context of human-computer interaction [6]. By understanding the dynamics of body language, one can interpret the underlying emotions, intentions, and attitudes of users.…”

mentioning

confidence: 99%

MAGIC-TBR: Multiview Attention Fusion for Transformer-based Bodily Behavior Recognition in Group Settings

Madan,

Jain,

Sharma

et al. 2023

Proceedings of the 31st ACM International Conference on Multimedia

View full text Add to dashboard Cite

Bodily behavioral language is an important social cue, and its automated analysis helps in enhancing the understanding of artificial intelligence systems. Furthermore, behavioral language cues are essential for active engagement in social agent-based user interactions. Despite the progress made in computer vision for tasks like head and body pose estimation, there is still a need to explore the detection of finer behaviors such as gesturing, grooming, or fumbling. This paper proposes a multiview attention fusion method named MAGIC-TBR that combines features extracted from videos and their corresponding Discrete Cosine Transform coefficients via a transformer-based approach. The experiments are conducted on the BBSI dataset and the results demonstrate the effectiveness of the proposed feature fusion with multiview attention. The code is available at: https://github.com/surbhimadan92/MAGIC-TBR CCS CONCEPTS• Computing methodologies → Machine learning; • Humancentered computing → Empirical studies in HCI.

show abstract

A Novel Machine Learning Based Two-Way Communication System for Deaf and Mute

et al. 2022

View full text Add to dashboard Cite

Deaf and mute people are an integral part of society, and it is particularly important to provide them with a platform to be able to communicate without the need for any training or learning. These people rely on sign language, but for effective communication, it is expected that others can understand sign language. Learning sign language is a challenge for those with no impairment. Another challenge is to have a system in which hand gestures of different languages are supported. In this manuscript, a system is presented that provides communication between deaf and mute (DnM) and non-deaf and mute (NDnM). The hand gestures of DnM people are acquired and processed using deep learning, and multiple language support is achieved using supervised machine learning. The NDnM people are provided with an audio interface where the hand gestures are converted into speech and generated through the sound card interface of the computer. Speech from NDnM people is acquired using microphone input and converted into text. The system is easy to use and low cost. The system is modular and can be enhanced by adding data to support more languages in the future. A supervised machine learning dataset is defined and created that provides automated multi-language communication between the DnM and NDnM people. It is expected that this system will support DnM people in communicating effectively with others and restoring a feeling of normalcy in their daily lives. The hand gesture detection accuracy of the system is more than 90% for most, while for certain scenarios, this is between 80% and 90% due to variations in hand gestures between DnM people. The system is validated and evaluated using a series of experiments.

show abstract

Attention-Augmented Electromagnetic Representation of Sign Language for Human-Computer Interaction in Deaf-and-Mute Community

Cited by 3 publications

References 3 publications

A Machine Learning Based Full Duplex System Supporting Multiple Sign Languages for the Deaf and Mute

A Machine Learning Based Full Duplex System Supporting Multiple Sign Languages for the Deaf and Mute

MAGIC-TBR: Multiview Attention Fusion for Transformer-based Bodily Behavior Recognition in Group Settings

A Novel Machine Learning Based Two-Way Communication System for Deaf and Mute

Contact Info

Product

Resources

About