Foreign Accent Conversion by Synthesizing Speech from Phonetic Posteriorgrams

Abstract

Methods for foreign accent conversion (FAC) aim to generate speech that sounds similar to a given non-native speaker but with the accent of a native speaker. Conventional FAC methods borrow excitation information (F0 and aperiodicity; produced by a conventional vocoder) from a reference (i.e., native) utterance during synthesis time. As such, the generated speech retains some aspects of the voice quality of the native speaker. We present a framework for FAC that eliminates the need for conventional vocoders (e.g., STRAIGHT, World) and therefore the need to use the native speaker's excitation. Our approach uses an acoustic model trained on a native speech corpus to extract speaker-independent phonetic posteriorgrams (PPGs), and then train a speech synthesizer to map PPGs from the non-native speaker into the corresponding spectral features, which in turn are converted into the audio waveform using a high-quality neural vocoder. At runtime, we drive the synthesizer with the PPG extracted from a native reference utterance. Listening tests show that the proposed system produces speech that sounds more clear, natural, and similar to the non-native speaker compared with a baseline system, while significantly reducing the perceived foreign accent of nonnative utterances.

Understanding the Effect of Voice Quality and Accent on Talker Similarity

“…al. [28] , which has been shown to produce higher ratings of acoustic quality and naturalness than traditional systems which use conventional vocoders such as STRAIGHT Figure 2: PPG-to-Mel conversion model [29] or World [30]. The system consists of three components: an acoustic model (AM) that extracts phonetic posteriorgrams (PPGs) from source utterances, a sequence-to-sequence (seq2seq) synthesizer that maps PPGs to Mel-spectrograms, and a WaveGlow vocoder that synthesizes speech waveforms from Mel-spectrograms.…”

“…from an ASR system) to estimate the posterior probability that each frame belongs to a set of pre-defined phonetic units (e.g., a phonetic posteriorgram, or PPG [27]). Once a PPG has been computed for each source and target frame in the corpus, the two can be paired in a many-to-many fashion based on the similarity between their respective PPGs [5,28].…”

“…Finally, a decoder LSTM, generates Mel-spectrograms using a location-sensitive attention mechanism [33]. In order to take PPGs as input, we replace the character-embedding layer with a PPG-embedding network [28]. This PPG-embedding layer consists of two fully connected hidden layers with ReLU nonlinearities.…”

“…This helps the model converge during training. The PPG to Mel-spectrogram model is shown in Figure 2; more details can be found in [28].…”

“…The WaveGlow models were trained using a batch size of 3 and the learning rate was 1×10 −4 for 650,000 iterations. We used the same set of parameters for the seq2seq model and the WaveGlow model as those reported in [28]. The AM was trained with Kaldi, and the other models were implemented in PyTorch and trained with the Adam optimizer [38].…”