UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022

Abstract

We present the UTokyo-SaruLab mean opinion score (MOS) prediction system submitted to VoiceMOS Challenge 2022. The challenge is to predict the MOS values of speech samples collected from previous Blizzard Challenges and Voice Conversion Challenges for two tracks: a main track for in-domain prediction and an out-of-domain (OOD) track for which there is less labeled data from different listening tests. Our system is based on ensemble learning of strong and weak learners. Strong learners incorporate several improvements to the previous finetuning models of self-supervised learning (SSL) models, while weak learners use basic machine-learning methods to predict scores from SSL features. In the Challenge, our system had the highest score on several metrics for both the main and OOD tracks. In addition, we conducted ablation studies to investigate the effectiveness of our proposed methods.

MooseNet: A trainable metric for synthesized speech with plda backend

“…We also use data augmentation and multi-task training described 2.2. Finally, we adapted contrastive loss [4] to boost ranking performance.…”

“…The MOS trainable metric mimics scores collected by human annotation studies which brings challenges in modeling score variance. Some listeners are more strict than others [2,4], and even a single listener adapts its judgments based on the quality of previously judged recordings.…”

“…The works [2,4] showed that it might be effective to use Listener Dependent (LD) modeling. LD modeling classifies listener 6 explicitly to explain the variance by conditioning it either on listener ID.…”

“…The challenge consists of two tracks. The size and diversity of the main track data allows building robust robust MOS predictor [3,4,5,6] and the OOD track is meant for testing MOS predictor under strong domain shift with little data for adaptation.…”

“…In our paper, we present fine-tuned methods for achieving state-of-the art results for MOS prediction on the VoiceMOS dataset including SSL baseline [3], contrastive loss [4], multi task learning using [7], and augmentation [8,9]. Our improved baseline achieved fourth place and third place on Main and OOD track respectively by using a pretrained SSL speech This work was supported by the Charles University GAUK grant number 40222, and the ERC grant number 101039303.…”