S2VC: A Framework for Any-to-Any Voice Conversion with Self-Supervised Pretrained Representations

Lin, Jheng-Hao; Lin, Yist Y.; Chien, Chung-Ming; Lee, Hung-yi

doi:10.48550/arxiv.2104.02901

Cited by 3 publications

(6 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The Summit supercomputer was utilized to train the protein LMs (pLMs), which required 5616 GPUs and a TPU Pod with up to 1024 cores. [102] performed numerical simulations, exploring the added value of three SSL models, notably (I) autoregressive predictive coding (APC), (II) contrastive predictive coding (CPC), and (III) wav2vec 2.0, in performing flexible classification and reliable recognition of datasets engaged in auto-regressive language modeling. Several any-to-any voice conversion (VC) methods have been proposed, like AUTOVC, AdaINVC, and FragmentVC.…”

Section: Auto-regressive Language Modelingmentioning

confidence: 99%

Consequential Advancements of Self-Supervised Learning (SSL) in Deep Learning Contexts

Abdulrazzaq,

Ramaha,

Hameed

et al. 2024

Mathematics

View full text Add to dashboard Cite

Self-supervised learning (SSL) is a potential deep learning (DL) technique that uses massive volumes of unlabeled data to train neural networks. SSL techniques have evolved in response to the poor classification performance of conventional and even modern machine learning (ML) and DL models of enormous unlabeled data produced periodically in different disciplines. However, the literature does not fully address SSL’s practicalities and workabilities necessary for industrial engineering and medicine. Accordingly, this thorough review is administered to identify these prominent possibilities for prediction, focusing on industrial and medical fields. This extensive survey, with its pivotal outcomes, could support industrial engineers and medical personnel in efficiently predicting machinery faults and patients’ ailments without referring to traditional numerical models that require massive computational budgets, time, storage, and effort for data annotation. Additionally, the review’s numerous addressed ideas could encourage industry and healthcare actors to take SSL principles into an agile application to achieve precise maintenance prognostics and illness diagnosis with remarkable levels of accuracy and feasibility, simulating functional human thinking and cognition without compromising prediction efficacy.

show abstract

Section: Auto-regressive Language Modelingmentioning

confidence: 99%

Consequential Advancements of Self-Supervised Learning (SSL) in Deep Learning Contexts

Abdulrazzaq,

Ramaha,

Hameed

et al. 2024

Mathematics

View full text Add to dashboard Cite

show abstract

“…In [29] the local and global style information are considered simultaneously. Selfsupervised speech representations are employed in [18] and [19] for voice conversion. Wang et al [16] used mutual information to measure the dependencies between speech representations.…”

Section: B Feature Disentanglement Based Voice Conversionmentioning

confidence: 99%

“…This reconstruction loss encourages well defined output spectrograms and ensures that the auto-encoder architecture does not loose too much information. It is also an essential part and a main objective for feature disentanglement-based any-to-any voice conversion methods [10], [11], [19].…”

Section: Loss Functionmentioning

confidence: 99%

“…To separate speaker timbre information from linguistic content as far as possible, many techniques have been proposed. These include the information constrained bottleneck layer [11], [13], phoneme transcription guidance [14], vector quantization [12], [15], [16], normalization techniques [10], [17] and selfsupervised speech representation [18], [19].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Any-to-Any Voice Conversion With Multi-Layer Speaker Adaptation and Content Supervision

Xu,

Shi,

Chen

et al. 2023

IEEE/ACM Trans. Audio Speech Lang. Process.

View full text Add to dashboard Cite

Any-to-any voice conversion can perform among arbitrary speakers with even just one single reference utterance. Many related studies have demonstrated that it can be effectively implemented by speech representation disentanglement. On the one hand, most existing solutions fuse the style representations into the content features in a global manner without considering the difference of distributions between them. On the other hand, in the any-to-any scenario, there is no effective method to ensure the consistency of the linguistic content without text transcription and additional information extracted from additional modules. To alleviate the above problems, in this paper, we propose a novel any-to-any voice conversion method, which we refer to as SACS-VC. It combines two principal modules, which are a) Style Adaptation and b) Content Supervision. Specifically, we rearrange the style representations according to the content distribution by using a temporal attention mechanism, to obtain finer-grained style timbre information for each individual content feature. Meanwhile, we associate the converted outputs and the source utterances directly to supervise the consistency of semantic content in an unsupervised manner. This can be achieved using contrastive learning based on the corresponding and the noncorresponding locations of content features. Additionally, our method can implement by using a non-parallel speech corpus without any pretraining. Experimental results demonstrate that our method outperforms the current state-of-the-art any-toany voice conversion systems in both objective and subjective evaluation settings.

show abstract

“…For example, FragmentVC applies the wav2vec 2.0 as the input of content encoders [13]. In S2VC [14], CPC [15] is used as the input of both the content encoder and the speaker encoder.…”

Section: Introductionmentioning

confidence: 99%

W2VC: WavLM representation based one-shot voice conversion with gradient reversal distillation and CTC supervision

Huang,

Wang,

Yang

et al. 2023

J AUDIO SPEECH MUSIC PROC.

View full text Add to dashboard Cite

Non-parallel data voice conversion (VC) has achieved considerable breakthroughs due to self-supervised pre-trained representation (SSPR) being used in recent years. Features extracted by the pre-trained model are expected to contain more content information. However, in common VC with SSPR, there is no special implementation to remove speaker information in the content representation extraction by SSPR, which prevents further purification of the speaker information from SSPR representation. Moreover, in conventional VC, Mel-spectrogram is often selected as the reconstructed acoustic feature, which is not consistent with the input of the content encoder and results in some information lost. Motivated by the above, we proposed W2VC to settle the issues. W2VC consists of three parts: (1) We reconstruct feature from WavLM representation (WLMR) that is more consistent with the input of content encoder; (2) Connectionist temporal classification (CTC) is used to align content representation and text context from phoneme level, content encoder plus gradient reversal layer (GRL) based speaker classifier are used to remove speaker information in the content representation extraction; (3) WLMR-based HiFi-GAN is trained to convert WLMR to waveform speech. VC experimental results show that GRL can purify well the content information of the self-supervised model. The GRL purification and CTC supervision on the content encoder are complementary in improving the VC performance. Moreover, the synthesized speech using the WLMR retrained vocoder achieves better results in both subjective and objective evaluation. The proposed method is evaluated on the VCTK and CMU databases. It is shown the method achieves 8.901 in objective MCD, 4.45 in speech naturalness, and 3.62 in speaker similarity of subjective MOS score, which is superior to the baseline.

show abstract

S2VC: A Framework for Any-to-Any Voice Conversion with Self-Supervised Pretrained Representations

Cited by 3 publications

References 22 publications

Consequential Advancements of Self-Supervised Learning (SSL) in Deep Learning Contexts

Consequential Advancements of Self-Supervised Learning (SSL) in Deep Learning Contexts

Any-to-Any Voice Conversion With Multi-Layer Speaker Adaptation and Content Supervision

W2VC: WavLM representation based one-shot voice conversion with gradient reversal distillation and CTC supervision

Contact Info

Product

Resources

About