S2VC: A Framework for Any-to-Any Voice Conversion with Self-Supervised Pretrained Representations

Lin, Jheng-Hao; Lin, Yist Y.; Chien, Chung-Ming; Lee, Hung-yi

doi:10.21437/interspeech.2021-1356

Cited by 37 publications

(12 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…One possible reason is that in the A2A VC setting, modern S3Rs still fail to disentangle content, such that the synthesizer preserves too much speaker information. Another reason may be that a jointly trained speaker encoder [10] is essential for S3R-based VC.…”

Section: Results On Different Tasksmentioning

confidence: 99%

“…ASR+TTS [4] was the seq2seq+non-AR vocoder baseline in VCC2020. S2VC [10] is the STOA system for A2A VC. The results are shown in Table 2.…”

Section: Comparing With Top Systems Using Subjective Evaluationmentioning

confidence: 99%

“…Therefore, researchers have resorted to unsupervised or the so-called selfsupervised speech representation (S3R) learning paradigm, where a large-scale unlabeled data is used to learn rich, compact speech representations. S3Rs have been applied to any-to-one VC [7], manyto-many VC [8], any-to-any VC [9,10] and cross-lingual VC [11].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

S3PRL-VC: Open-source Voice Conversion Framework with Self-supervised Speech Representations

Huang¹,

Yang²,

Hayashi³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

This paper introduces S3PRL-VC, an open-source voice conversion (VC) framework based on the S3PRL toolkit. In the context of recognition-synthesis VC, self-supervised speech representation (S3R) is valuable in its potential to replace the expensive supervised representation adopted by state-of-the-art VC systems. Moreover, we claim that VC is a good probing task for S3R analysis. In this work, we provide a series of in-depth analyses by benchmarking on the two tasks in VCC2020, namely intra-/cross-lingual any-to-one (A2O) VC, as well as an any-to-any (A2A) setting. We also provide comparisons between not only different S3Rs but also top systems in VCC2020 with supervised representations. Systematic objective and subjective evaluation were conducted, and we show that S3R is comparable with VCC2020 top systems in the A2O setting in terms of similarity, and achieves state-of-the-art in S3R-based A2A VC. We believe the extensive analysis, as well as the toolkit itself, contribute to not only the S3R community but also the VC community. The codebase is now open-sourced 1 .

show abstract

Section: Results On Different Tasksmentioning

confidence: 99%

“…ASR+TTS [4] was the seq2seq+non-AR vocoder baseline in VCC2020. S2VC [10] is the STOA system for A2A VC. The results are shown in Table 2.…”

Section: Comparing With Top Systems Using Subjective Evaluationmentioning

confidence: 99%

See 1 more Smart Citation

S3PRL-VC: Open-source Voice Conversion Framework with Self-supervised Speech Representations

Huang¹,

Yang²,

Hayashi³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…To understand how Retriever models the content, the style, and the bipartite graph between them, we visualize the VQ codes and the decoder link attention map for a 2.5s-long test utterance in (Chou & Lee, 2019) 46.5% 1.86±0.10 2.26±0.14 FragmentVC (Lin et al, 2021b) 89.5% 3.43±0.12 3.54±0.15 S2VC (Lin et al, 2021a) 96.8% 3.18±0.12 3.36±0.15 Retriever 99.4% 3.44±0.13 3.84±0.14…”

Section: Visualization Of Content-style Representationmentioning

confidence: 99%

“…F.1 IMPLEMENTATION DETAILSFor the tokenization module, we followLin et al (2021a), using s3prl toolkit 1 to extract the CPC feature, and use a depth-wise convolutional layer with kernel size 15 after the CPC feature. The depth-wise convolutional layer is trainable, while the CPC model is fixed during training.…”

mentioning

confidence: 99%

Retriever: Learning Content-Style Representation as a Token-Level Bipartite Graph

Yin¹,

Ren²,

Luo³

et al. 2022

Preprint

View full text Add to dashboard Cite

This paper addresses the unsupervised learning of content-style decomposed representation. We first give a definition of style and then model the content-style representation as a token-level bipartite graph. An unsupervised framework, named Retriever, is proposed to learn such representations. First, a cross-attention module is employed to retrieve permutation invariant (P.I.) information, defined as style, from the input data. Second, a vector quantization (VQ) module is used, together with man-induced constraints, to produce interpretable content tokens. Last, an innovative link attention module serves as the decoder to reconstruct data from the decomposed content and style, with the help of the linking keys. Being modal-agnostic, the proposed Retriever is evaluated in both speech and image domains. The state-of-the-art zero-shot voice conversion performance confirms the disentangling ability of our framework. Top performance is also achieved in the part discovery task for images, verifying the interpretability of our representation. In addition, the vivid part-based style transfer quality demonstrates the potential of Retriever to support various fascinating generative tasks. Project page at https://ydcustc.github.io/retriever-demo/.

show abstract

FMFCC-A: A Challenging Mandarin Dataset for Synthetic Speech Detection

Zhang

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

S2VC: A Framework for Any-to-Any Voice Conversion with Self-Supervised Pretrained Representations

Cited by 37 publications

References 0 publications

S3PRL-VC: Open-source Voice Conversion Framework with Self-supervised Speech Representations

S3PRL-VC: Open-source Voice Conversion Framework with Self-supervised Speech Representations

Retriever: Learning Content-Style Representation as a Token-Level Bipartite Graph

FMFCC-A: A Challenging Mandarin Dataset for Synthetic Speech Detection

Contact Info

Product

Resources

About