Optimization of Cross-Lingual Voice Conversion With Linguistics Losses to Reduce Foreign Accents

Abstract

Cross-lingual voice conversion (XVC) transforms the speaker identity of a source speaker to that of a target speaker who speaks a different language. Due to the intrinsic differences between languages, the converted speech may carry an unwanted foreign accent. In this paper, we first investigate the intelligibility of the converted speech and confirm the performance degradation caused by the accent/intelligibility issue. With the goal of generating native-sounding speech, this paper further proposes a novel training scheme with two additional linguistic losses for speech waveform generation: 1) a frame-wise phonetic content loss derived from bottleneck features, and 2) an automatic speech recognition loss on characters. Experiments were conducted between English and Mandarin Chinese conversions. The experimental results confirmed that the generated speech sounds more natural with the proposed linguistic losses and the proposed solution significantly improves speech intelligibility.

Keystroke Dynamics as a Language Profiling Tool: Identifying Mother Tongue of Unknown Internet Users

“…They proposed a triplet training scheme composed of an anchor, a positive, and a negative sample to cover unseen cases. A similar problem was dealt with by Zhou et al [14], who tried to improve pronunciation when converting speech into another language, using cross-lingual voice conversion techniques. Finally, Vaswani et al [15] proposed a new simple network architecture, the "Transformer", for translating one language to another and achieved outstanding results.…”