Decoding lip language using triboelectric sensors with deep learning

Abstract

Lip language is an effective method of voice-off communication in daily life for people with vocal cord lesions and laryngeal and lingual injuries without occupying the hands. Collection and interpretation of lip language is challenging. Here, we propose the concept of a novel lip-language decoding system with self-powered, low-cost, contact and flexible triboelectric sensors and a well-trained dilated recurrent neural network model based on prototype learning. The structural principle and electrical properties of the flexible sensors are measured and analysed. Lip motions for selected vowels, words, phrases, silent speech and voice speech are collected and compared. The prototype learning model reaches a test accuracy of 94.5% in training 20 classes with 100 samples each. The applications, such as identity recognition to unlock a gate, directional control of a toy car and lip-motion to speech conversion, work well and demonstrate great feasibility and potential. Our work presents a promising way to help people lacking a voice live a convenient life with barrier-free communication and boost their happiness, enriches the diversity of lip-language translation systems and will have potential value in many applications.

Morphological Engineering of Sensing Materials for Flexible Pressure Sensors and Artificial Intelligence Applications

“…Lu et al designed a decoding lip language system based on TENG with deep learning (Fig. 20 d-e) [ 59 ]. The related confusion matrix can be used to match words with similar lip motion signals and guide the further improvement of TENG (Fig.…”

“…Copyright (2021) Springer Nature. d-g lip-language decoding system [ 59 ]. Copyright (2022) The Authors.…”

A non-battery pressure detection and communication system for basketball game referee based on piezoelectric devices

“…In the bottom view, four positions to place the sensing patches can be viewed, which are the internal part below the toes, front soles under the arch of the foot, back soles under the arch of the foot, and the internal part of the heel. The interface module contains the energy harvesting module, communication chip, and circuit lines, which can be minimized to very small size and attached to the human skin easier by the state-of-art designs such as triboelectric sensors and adhesive technology [40,41]. All the electrons are connected to the common-ground and flow through the chain to form a piezo electrical potential by resuming the balanced state.…”

Pushing the limits of remote RF sensing by reading lips under the face mask

“…A related work is presented in ref. 23 , where authors use the structural principle and electrical properties of the flexible triboelectric sensors to decode the lip movements. The sensors are placed inside a pseudo mask where the lips remained exposed to make them clearly visible.…”