Two-stage visual speech recognition for intensive care patients

Laux, Hendrik; Hallawa, Ahmed; Assis, Julio Cesar Sevarolli; Schmeink, Anke; Märtin, Lukas; Peine, Arne

doi:10.1038/s41598-022-26155-5

Cited by 3 publications

(1 citation statement)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…SR using audio-visual signals, which involves combining information from both audio and visual cues to identify or verify the speaker's identity, has gained increasing attention in recent years [4], [5]. While SR using audio-visual signals has the potential to improve the accuracy and reliability of SR systems [6]- [8], it is important to consider the limitations of this technology [9]. These limitations include lighting conditions, environmental noise, occlusions, facial expressions, privacy concerns, equipment costs, and algorithmic bias.…”

Section: Introductionmentioning

confidence: 99%

Privacy-Preserving Speaker Recognition Using Radars for Context Estimation in Future Multi-Modal Hearing Assistive Technologies

Farooq,

Ge,

Qayyum

et al. 2023

2023 IEEE International Radar Conference (RADAR)

View full text Add to dashboard Cite

Speaker recognition (SR) from speech can help determine the environmental context in multi-talker conversational scenarios to enable the design of context-aware multi-modal hearing assistive technology. In this paper, we argue that the use of wireless sensors such as radars can offer many benefits over conventional audio and visual sensors, such as not being afflicted by privacy and environmental issues, e.g., improper lighting, environmental noise, and potential security concerns of audio and video channels. Radar is relatively less explored and has many advantages over other contactless approaches, such as being more compact compared to RFID and having a better range and resolution than ultrasound and microwave sensors. To this end, we propose the use of ultrawideband radar coupled with a deep learning model for SR from silent speech to enable the design of future context-aware multimodal hearing assistive technology. We collected a dataset from five individuals with origins in Europe, Asia, and the United Kingdom. We obtained an average performance of approximately 82% in recognising an unknown person from a set of known people. This demonstrates that the radar has good potential to be used for privacy-preserving SR in multi-talker environments where audio-visual and other contactless techniques have limited capabilities.

show abstract

Section: Introductionmentioning

confidence: 99%

Privacy-Preserving Speaker Recognition Using Radars for Context Estimation in Future Multi-Modal Hearing Assistive Technologies

Farooq,

Ge,

Qayyum

et al. 2023

2023 IEEE International Radar Conference (RADAR)

View full text Add to dashboard Cite

show abstract

LITEVSR: Efficient Visual Speech Recognition by Learning from Speech Representations of Unlabeled Data

Laux,

Mededovic,

Hallawa

et al. 2024

ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

HNet: A deep learning based hybrid network for speaker dependent visual speech recognition

Chandrabanshi,

Domnic

2024

International Journal of Hybrid Intelligent Systems

View full text Add to dashboard Cite

Visual Speech Recognition (VSR) is a popular area in computer vision research, attracting interest for its ability to precisely analyze lip motion and seamlessly convert them into textual representation. VSR systems leverage visual features to augment the understanding of automated speech and predict text. VSR finds various applications, including enhancing speech recognition in scenarios with degraded acoustic signals, aiding individuals with hearing impairments, bolstering security by reducing reliance on text based passwords, facilitating biometric authentication for liveness detection, and enabling underwater communications. Despite the various techniques proposed for improving the resilience and precision of automatic speech recognition, VSR still has challenges like homophones of words, gradient descent issues with varying sequence lengths, and lip reading demands accounting for short and long range correlations between consecutive video frames. We have proposed a hybrid network (HNet) with multilayered three dimensional dilated convolution neural network (3D-CNN). The spatio-temporal feature extraction process will be facilitated by dilated 3D-CNN. HNet integrates two bidirectional recurrent neural networks (BiGRU and BiLSTM) to process the feature sequences bidirectionally to establish the temporal relationship. The fusion of BiGRU-BiLSTM capabilities allows the model to process feature sequences more comprehensively and effectively. The proposed work focuses on face based biometric authentication for liveness detection using the VSR model to boost security against face spoofing. The existing face based biometric systems are widely used for individual authentication and verification but are still vulnerable to 3D masks and adversarial attacks. The VSR system can be added to existing face based verification systems as a second level authentication technique to identify a person with liveness. The working ideology of the VSR system will be based on the challenge response technique, where a person has to pronounce the passcode silently displayed on the screen. The VSR model assesses its effectiveness using word error rate (WER), which matches the pronounced passcode to the one presented on the screen. Overall, the proposed work aims to enhance the accuracy of VSR so that it can be combined with existing face based authentication systems. The proposed system outperforms the existing VSR system and obtained 1.3% WER. The significance of the proposed hybrid model is that it efficiently captures temporal dependencies, enhancing context embedding, improving robustness to input variability, reducing information loss, and enhancing performance and accuracy in modeling and analyzing passcode pronunciation patterns.

show abstract

Two-stage visual speech recognition for intensive care patients

Cited by 3 publications

References 25 publications

Privacy-Preserving Speaker Recognition Using Radars for Context Estimation in Future Multi-Modal Hearing Assistive Technologies

Privacy-Preserving Speaker Recognition Using Radars for Context Estimation in Future Multi-Modal Hearing Assistive Technologies

LITEVSR: Efficient Visual Speech Recognition by Learning from Speech Representations of Unlabeled Data

HNet: A deep learning based hybrid network for speaker dependent visual speech recognition

Contact Info

Product

Resources

About