Machine Learning Methods for Automatic Silent Speech Recognition Using a Wearable Graphene Strain Gauge Sensor

Abstract

Silent speech recognition is the ability to recognise intended speech without audio information. Useful applications can be found in situations where sound waves are not produced or cannot be heard. Examples include speakers with physical voice impairments or environments in which audio transference is not reliable or secure. Developing a device which can detect non-auditory signals and map them to intended phonation could be used to develop a device to assist in such situations. In this work, we propose a graphene-based strain gauge sensor which can be worn on the throat and detect small muscle movements and vibrations. Machine learning algorithms then decode the non-audio signals and create a prediction on intended speech. The proposed strain gauge sensor is highly wearable, utilising graphene’s unique and beneficial properties including strength, flexibility and high conductivity. A highly flexible and wearable sensor able to pick up small throat movements is fabricated by screen printing graphene onto lycra fabric. A framework for interpreting this information is proposed which explores the use of several machine learning techniques to predict intended words from the signals. A dataset of 15 unique words and four movements, each with 20 repetitions, was developed and used for the training of the machine learning algorithms. The results demonstrate the ability for such sensors to be able to predict spoken words. We produced a word accuracy rate of 55% on the word dataset and 85% on the movements dataset. This work demonstrates a proof-of-concept for the viability of combining a highly wearable graphene strain gauge and machine leaning methods to automate silent speech recognition.

Research on Throat Speech Signal Detection Based on a Flexible Graphene Piezoresistive Sensor

“…Although the process is relatively simple, researchers cannot ensure that the thickness of each batch of materials is consistent during operation. The performance of materials varies with their thickness . Here, we choose a chemical vapor deposition (CVD) method, which can control the time and temperature parameters to prepare graphene, which can better control the number of layers of graphene and ensure that each batch of sensors has similar properties.…”

“…These are then able to identify local features by examining the relationship between adjacent data points of the input results in the formation of spatial features. At the backward pass stage, these filters are automatically optimized via backpropagating the prediction error of the forward pass . The network also includes two fully connected layers, and the role of these two layers is to fuse the feature map information of each channel.…”

“…The performance of materials varies with their thickness. 33 Here, we choose a chemical vapor deposition (CVD) method, which can control the time and temperature parameters to prepare graphene, which can better control the number of layers of graphene and ensure that each batch of sensors has similar properties. The preparation process of the graphene film is shown in Figure 1a.…”

“…At the backward pass stage, these filters are automatically optimized via backpropagating the prediction error of the forward pass. 33 The network also includes two fully connected layers, and the role of these two layers is to fuse the feature map information of each channel. With a Relu activation function, the first fully connected layer acts as a dimensionality reduction, while the second fully connected layer increases the dimensionality to the original input dimension.…”

Speech Recognition Using Intelligent Piezoresistive Sensor Based on Polystyrene Sphere Microstructures

“…More recently, a graphene‐based wearable sensor was attached to human throat to classify the words by using the neural network with an accuracy of 55%. [ 39 ] In this case, a more accurate system should be developed for speech recognition by wearable sensors.…”