CELIP: Ultrasonic-based Lip Reading with Channel Estimation Approach for Virtual Reality Systems

Abstract

We developed an ultrasonic-based silent speech interface for Virtual Reality (VR). As more and more customized devices are proposed to enhance the immersion and experience of VR, our system can be used to improve the capability of interactions between users and the systems, while retaining the possibilities of using various customized devices and avoiding some limitations of traditional speech recognition. By employing the channel estimation techniques with ultrasonic waves, we can derive movement characteristics of users' lips, which can be used to fine-tune existing speech recognition models and augmented by vast open-sourced speech datasets. Moreover, we use the speech interface to guide the initialization of customized models for new users, so that they can easily have the access to our system. A two-stage experiment has been conducted and the results show that our system can achieve 90.8% commandlevel accuracy and 1.3% word-error-rate in sentence-level accuracy. CCS CONCEPTS• Human-centered computing → Interaction techniques.

Microwave Speech Recognizer Empowered by a Programmable Metasurface

“…Obviously, such contactbased approaches are oftentimes inconvenient and, moreover, incompatible with largescale deployment in our daily lives. Similar limitations apply to contactless radar-like approaches based on the emission and reception of acoustic 8,9 or electromagnetic 10,11 waves in the very close proximity (a few centimeters) of the speaker's face. A popular contactless technique for speech recognition that can operate remotely uses optical image sequences as secondary information carrier to recognize speech by analyzing lip or face motion 12,13 .…”

Electromyogram‐Based Lip‐Reading via Unobtrusive Dry Electrodes and Machine Learning Methods

“…Lip-reading also provides an effective human-machine interface that uses natural speech for interactions with smart machines, [5,6] including robotics, [7] prosthetics, [8] computers, [9] and even augmented reality (AR)/virtual reality (VR) environments. [10,11] Silent speech interfaces, where no acoustic sound is produced, are often in demand when privacy or a quiet environment is desired, such as in public areas or hospitals. To realize lip-reading, camera-based visual solutions have been extensively explored to capture the visual features of lip movements.…”

Decoding silent speech commands from articulatory movements through soft magnetic skin and machine learning

“…Contactless approaches are mainly explored through camera-based visual signals, [9][10][11][12][13][14][15] ultrasound signals. [16][17][18][19][20][21][22][23] Camerabased visual solutions require external video tracking devices, and users must remain within the camera's line of sight. Despite efforts to develop compact shoulder-mounted devices 9 to enhance portability, visual solutions still face challenges in terms of lighting conditions and angles between users and cameras, thereby limiting their practicality.…”