Optimizing the Ultrasound Tongue Image Representation for Residual Network-Based Articulatory-to-Acoustic Mapping

Abstract

Within speech processing, articulatory-to-acoustic mapping (AAM) methods can apply ultrasound tongue imaging (UTI) as an input. (Micro)convex transducers are mostly used, which provide a wedge-shape visual image. However, this process is optimized for the visual inspection of the human eye, and the signal is often post-processed by the equipment. With newer ultrasound equipment, now it is possible to gain access to the raw scanline data (i.e., ultrasound echo return) without any internal post-processing. In this study, we compared the raw scanline representation with the wedge-shaped processed UTI as the input for the residual network applied for AAM, and we also investigated the optimal size of the input image. We found no significant differences between the performance attained using the raw data and the wedge-shaped image extrapolated from it. We found the optimal pixel size to be 64 × 43 in the case of the raw scanline input, and 64 × 64 when transformed to a wedge. Therefore, it is not necessary to use the full original 64 × 842 pixels raw scanline, but a smaller image is enough. This allows for the building of smaller networks, and will be beneficial for the development of session and speaker-independent methods for practical applications. AAM systems have the target application of a “silent speech interface”, which could be helpful for the communication of the speaking-impaired, in military applications, or in extremely noisy conditions.

Is Dynamic Time Warping of speech signals suitable for articulatory signal comparison using ultrasound tongue images?

“…The use of articulatory information in speech technology is less mature than standard speech recognition or speech synthesis; those methods using articulatory signals are currently at the basic research level and do not yet have applications for everyday people. Most related research deals with the way how articulatory information can be used to extend speech technology, for example as input or output of the system, like articulatory-to-acoustic mapping [9], [15], [16] or acoustic-toarticulatory inversion [17], [27].…”

“…SSI systems represent a revolutionary direction in speech technology, where silent articulatory movements are captured by some device and from this speech is automatically generated while the original speaker does not make a sound [3]. In most previous research on SSI, only a few speakers have been studied [3]- [9]. Although the results of these studies are encouraging, further research is needed to develop session-and speakerindependent SSI systems [10].…”

“…The ultrasound tongue images were used as 8-bit grayscale pixels in the raw ultrasound form of the "Micro" system. The images, originally 64x842 pixels, were resized to 64x128 pixels as this does not cause significant loss of information [9]. The ultrasound image is relatively redundant and can therefore be compressed efficiently, which can be an advantage in subsequent processing, as we only need to work with data of smaller dimensions.…”

Improvements of Silent Speech Interface Algorithms

“…2. 1 The layers of the 2D and 3D CNNs in the Keras implementation, along with their most important parameters.…”

“…In addition to using the VAD implementation available from WebRTC [1], we trained a CNN to perform VAD directly from ultrasound images. volved using a simple frame-by-frame approach, where a single image was used as input and a 2D-CNN was applied to classify each frame as either silence or speech(SI/SP).…”