Phoneme Embedding and its Application to Speech Driven Talking Avatar Synthesis

Abstract

Word embedding has made great achievements in many natural language processing tasks. However, the attempt to apply word embedding to the field of speech got few breakthroughs. The reason is that word vectors mainly contain semantic and syntactic information. Such high level features are difficult to be directly incorporated in speech related tasks compared to acoustic or phoneme related features. In this paper, we investigate the method for phoneme embedding to generate phoneme vectors carrying acoustic information for speech related tasks. One-hot representations of phoneme labels are fed into embedding layer to generate phoneme vectors that are then passed through bidirectional long short-term memory (BLSTM) recurrent neural network to predict acoustic features. Weights in embedding layer are updated through backpropagation during training. Analyses indicate that phonemes with similar acoustic pronunciations are close to each other in cosine distance in the generated phoneme vector space, and tend to be in the same category after k-means clustering. We evaluate the phoneme embedding by applying the generated phoneme vector into speech driven talking avatar synthesis. Experimental results indicate that adding phoneme vector as features can achieve 10.2% relative improvement in objective test.

ODSQA: Open-Domain Spoken Question Answering Dataset

“…The network details are listed as follow: pingyin-token embedding size 6, filter size 3x6 and numbers of filters 100. Different from [41] using one-hot vector, we choose distributed representation vectors to represent subword units. The experimental results with and without the proposed sub-word unit approach are in Table 4.…”

Sound Analogies with Phoneme Embeddings

“…In that work, a neural encoder-decoder model (Cho et al, 2014;Sutskever et al, 2014) was trained to perform a transformation of words from their citation forms to a 'target' inflected form and, after training, the vowels in the embedding layer of the long-short term memory (LSTM) neural model trained for Finnish were found to clearly group themselves according to known harmony patterns in the language. Li et al (2016) take advantage of phoneme transcriptions in a neural speech synthesis application, showing improvements on this task and indicating that similar phonemes in a bidirectional LSTM (Bi-LSTM) embedding layer map close to each other. More closely related to the current work, Dunbar et al (2015) investigate how well phonetic feature representations in English align with vector representations learned from local contexts of sound occurrence using both a neural language model and also a matrix factorization model.…”

Using Phoneme Representations to Build Predictive Models Robust to ASR Errors

“…Few existing works studied phoneme embeddings. Li et al [13] explored the application of phoneme embeddings for the task of speech-driven talking avatar synthesis to create more realistic and expressive visual gestures. Silfverberg et al [30] proposed an approach to learn phoneme embeddings that can be used to perform phonological analogies.…”

“…We use word embeddings (300 dimensions) trained on Wikipedia from GloVe [24]. The character embeddings (20-dimensional) are randomly initialized and trainable, whereas the pre-trained phoneme embeddings are not trainable 13 , except the randomly initialized ones (rnd model).…”