Direct Speech Generation for a Silent Speech Interface based on Permanent Magnet Articulography

et al. 2017

Self Cite

Abstract. Silent Speech Interfaces (SSIs) are alternative assistive speech technologies that are capable of restoring speech communication for those individuals who have lost their voice due to laryngectomy or diseases affecting the vocal cords. However, many of these SSIs are still deemed as impractical due to a high degree of intrusiveness and discomfort, hence limiting their transition to outside of the laboratory environment. We aim to address the hardware challenges faced in developing a practical SSI for post-laryngectomy speech rehabilitation. A new Permanent Magnet Articulography (PMA) system is presented which fits within the palatal cavity of the user's mouth, giving unobtrusive appearance and high portability. The prototype is comprised of a miniaturized circuit constructed using commercial off-the-shelf (COTS) components and is implemented in the form of a dental retainer, which is mounted under roof of the user's mouth and firmly clasps onto the upper teeth. Preliminary evaluation via speech recognition experiments demonstrates that the intraoral prototype achieves reasonable word recognition accuracy and is comparable to the external PMA version. Moreover, the intraoral design is expected to improve on its stability and robustness, with a much improved appearance since it can be completely hidden inside the user's mouth. [8]. To date, a number of SSIs have been proposed in an attempt to extract non-acoustic information generated during speech production and reproduce audible speech using different sensing modalities, such as measuring electrical activities of the brain [9][10][11] or the articulator muscles [12][13][14], or by capturing movements of the speech articulators themselves [3,5,8,[15][16][17][18][19]. A comprehensive summary on different SSIs technologies were presented in [8]. Because of their unique feature, SSIs can also be deployed in acoustically challenging environment or where privacy/confidentially is desirable, and not limited to its use as a communication aid for speech impaired individuals. KeywordsDespite the attractive attributes of SSIs, there are still challenges in the form of hardware (e.g. portability, lightweight, unobtrusiveness and wearability) and processing software (e.g. efficiency, robustness and intelligibility speech generation). Preliminary discussions on the influential factors affecting the SSIs' implementation were presented in [8], based upon criteria such as ability to operate in silence and noisy environments, usability by laryngectomees, issue of invasiveness market reediness and cost.In the present work we employ the Permanent Magnet Articulography (PMA), which is a type SSI that is based on sensing the changes in the magnetic field generated by a set of permanent magnet markers attached onto the vocal apparatus (i.e. lips and tongue) during speech articulation by using an array of magnetic sensors located around the mouth [1,3]. Although PMA shares some similarities with Electromagnetic Articulography (EMA) [5,17], it does not explicitly provide the ...

Section: Resultsmentioning

confidence: 99%

Towards an Intraoral-Based Silent Speech Restoration System for Post-laryngectomy Voice Replacement

et al. 2017

Self Cite

“…Apart from that, another reason the MCD results are worse on the Arctic database is the limitations of the current PMA device for modelling some areas of the vocal tract (e.g. sounds articulated at the back of the mouth) [20,21], as discussed below. Since the Arctic sentences contain more phones articulated in those areas than the digits vocabulary, this harms the overall reconstruction performance achieved on the Arctic database.…”

Section: Results On the Tidigits Databasementioning

confidence: 99%

“…The results are presented as a function of the number of mixture components in the MFA model (i.e. K in (20)) and the length of the sliding window used to extract the PMA parameter vectors. As can be seen, results greatly improve when more mixture components and longer windows are used for modelling the PMA-to-acoustic mapping.…”

Section: Results On the Tidigits Databasementioning

confidence: 99%

“…Finally, by substituting the expression of the conditional distribution p(y|x t ) in (20) into (19), we reach the following expression for the MMSE estimation of the speech parameter vectors,…”

Section: Conversion Phasementioning

confidence: 99%

“…The audio signals are represented in this work as 25 Mel-Frequency Cepstral Coefficients (MFCCs) [16] obtained at the same frame rate as that for PMA. Neither F 0 nor voicing information are extracted from the audio signals because of the limited ability of PMA to model this aspect of speech articulation [20]. Rather, the audio signals are re-synthesised without voicing as whispered speech.…”

Section: Feature Extractionmentioning

confidence: 99%

See 2 more Smart Citations

Voice Restoration After Laryngectomy Based on Magnetic Sensing of Articulator Movement and Statistical Articulation-to-Speech Conversion

et al. 2017

Self Cite

ReuseUnless indicated otherwise, fulltext items are protected by copyright with all rights reserved. The copyright exception in section 29 of the Copyright, Designs and Patents Act 1988 allows the making of a single copy solely for the purpose of non-commercial research or private study within the limits of fair dealing. The publisher or other rights-holder may allow further reproduction and re-use of this version -refer to the White Rose Research Online record for this item. Where records identify the publisher as the copyright holder, users can verify any specific terms of use on the publisher's website. TakedownIf you consider content in White Rose Research Online to be in breach of UK law, please notify us by emailing eprints@whiterose.ac.uk including the URL of the record and the reason for the withdrawal request. Abstract. In this work, we present a silent speech system that is able to generate audible speech from captured movement of speech articulators. Our goal is to help laryngectomy patients, i.e. patients who have lost the ability to speak following surgical removal of the larynx most frequently due to cancer, to recover their voice. In our system, we use a magnetic sensing technique known as Permanent Magnet Articulography (PMA) to capture the movement of the lips and tongue by attaching small magnets to the articulators and monitoring the magnetic field changes with sensors close to the mouth. The captured sensor data is then transformed into a sequence of speech parameter vectors from which a time-domain speech signal is finally synthesised. The key component of our system is a parametric transformation which represents the PMA-to-speech mapping. Here, this transformation takes the form of a statistical model (a mixture of factor analysers, more specifically) whose parameters are learned from simultaneous recordings of PMA and speech signals acquired before laryngectomy. To evaluate the performance of our system on voice reconstruction, we recorded two PMA-and-speech databases with different phonetic complexity for several non-impaired subjects. Results show that our system is able to synthesise speech that sounds as the original voice of the subject and also is intelligible. However, more work still need to be done to achieve a consistent synthesis for phonetically-rich vocabularies.

Integrating User-Centred Design in the Development of a Silent Speech Interface Based on Permanent Magnetic Articulography

et al. 2015

Self Cite

A new wearable silent speech interface (SSI) based on Permanent Magnetic Articulography (PMA) was developed with the involvement of end users in the design process. Hence, desirable features such as appearance, portability, ease of use and light weight were integrated into the prototype. The aim of this paper is to address the challenges faced and the design considerations addressed during the development. Evaluation on both hardware and speech recognition performances are presented here. The new prototype shows a comparable performance with its predecessor in terms of speech recognition accuracy (i.e. ~95% of word accuracy and ~75% of sequence accuracy), but significantly improved appearance, portability and hardware features in terms of miniaturization and cost.