SpeeChin

Abstract

This paper presents SpeeChin, a smart necklace that can recognize 54 English and 44 Chinese silent speech commands. A customized infrared (IR) imaging system is mounted on a necklace to capture images of the neck and face from under the chin. These images are first pre-processed and then deep learned by an end-to-end deep convolutional-recurrent-neural-network (CRNN) model to infer different silent speech commands. A user study with 20 participants (10 participants for each language) showed that SpeeChin could recognize 54 English and 44 Chinese silent speech commands with average cross-session accuracies of 90.5% and 91.6%, respectively. To further investigate the potential of SpeeChin in recognizing other silent speech commands, we conducted another study with 10 participants distinguishing between 72 one-syllable nonwords. Based on the results from the user studies, we further discuss the challenges and opportunities of deploying SpeeChin in real-world applications.

LipLearner: Customizable Silent Speech Interactions on Mobile Devices

“…Lipreading is a technology that utilizes a camera to visually capture movement around the mouth and interpret speech from the image sequence. HCI researchers have proposed to use devices such as smartphones [45,58] and wearable cameras [6,34,69] to provide mobile silent speech interaction, as well as multimodal approaches such as using silent speech to facilitate eye-gaze-based selection [57].…”

“…Additionally, there is a lack of a practical activating method to initiate silent speech input. Previous methods such as offline segmentation [6,34,69] or trigger buttons [45,52] are not feasible for hands-free real-time interactions, and MOD-based methods can be vulnerable to misactivations [57,58]. We propose a novel few-shot transfer learning paradigm to enable customizable silent speech commands.…”

“…Detecting and segmenting the user's silent speech has been challenging in real-time lipreading. Previous researchers have proposed to activate the recognition algorithm by using the opening degree of the mouth to identify silent speech [57,58,69]. However, this approach is prone to misactivation because it can be easily confused when the user is talking to others or unintentionally opens their mouth.…”

“…First, the data collection process should minimize the effort for new users to get started with. However, previous approaches to SSIs, not limited to lipreading-based approaches, adopt a train-from-scratch model that requires collecting hundreds of samples from real users [34,57,58,69], leading to excessive mental and physical user burden. Second, such data collected intensively in controlled laboratory environments causes a biased model, which can be sensitive to even minor changes in factors such as lighting, face orientations, and postures, yet there is little discussion on the model's ability to generalize to unseen environments.…”

Microwave Speech Recognizer Empowered by a Programmable Metasurface

“…Therefore, a wide variety of indirect secondary carriers of information about voice commands have been explored. Many of these techniques achieve high accuracy at the price of being highly invasive because they rely on placing sensors (e.g., magnetic, [ 2 ] surface electromyographic, [ 3 ] infrared, [ 4 ] electropalatographic, [ 5 ] electromagnetic [ 6 , 7 ] ) directly on the human's body to detect subtle vibrations that are correlated with the speech production. Obviously, such contact‐based approaches are oftentimes inconvenient and, moreover, incompatible with large‐scale deployment in our daily lives.…”