Automatic Pronunciation Assessment using Self-Supervised Speech Representation Learning

Kim, Eesung; Jeon, Jae Hyun; Seo, Hyeji; Kim, Hoon

doi:10.48550/arxiv.2204.03863

Cited by 5 publications

(6 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As can be seen in Table 3, the system with the setting of "SSL + ASR-based", i.e., SSL features with forced-alignment, achieves the best performance, followed by the proposed approach and then the other two approaches. The recent work "SSL & Text" [28] developed a scorer by modeling the SSL speech representations and prompt text information as two separate streams. Note that different sizes and types of SSL models were tried and the best-reported performance comes from the non-native ASR fine-tuned HuBERT Large.…”

Section: Experimental Results In the "Read Aloud" Scenariomentioning

confidence: 99%

An ASR-free Fluency Scoring Approach with Self-Supervised Learning

Liu¹,

Fu²,

Tian³

et al. 2023

Preprint

View full text Add to dashboard Cite

A typical fluency scoring system generally relies on an automatic speech recognition (ASR) system to obtain time stamps in input speech for either the subsequent calculation of fluency-related features or directly modeling speech fluency with an end-to-end approach. This paper describes a novel ASR-free approach for automatic fluency assessment using self-supervised learning (SSL). Specifically, wav2vec2.0 is used to extract frame-level speech features, followed by K-means clustering to assign a pseudo label (cluster index) to each frame. A BLSTM-based model is trained to predict an utterance-level fluency score from frame-level SSL features and the corresponding cluster indexes. Neither speech transcription nor time stamp information is required in the proposed system. It is ASR-free and can potentially avoid the ASR errors effect in practice. Experimental results carried out on non-native English databases show that the proposed approach significantly improves the performance in the "open response" scenario as compared to previous methods and matches the recently reported performance in the "read aloud" scenario.

show abstract

Section: Experimental Results In the "Read Aloud" Scenariomentioning

confidence: 99%

An ASR-free Fluency Scoring Approach with Self-Supervised Learning

Liu¹,

Fu²,

Tian³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Table 4 compares the proposed approach with various other approaches using different datasets such as Spee-chocean762, TIMIT, LibriSpeech, and more. Latest models such as HuBERT [35] and Wav2Vec2 [36] were also compared. It should be noted that our model was not fine-tuned whereas all other models were fine-tuned before getting these results.…”

Section: Resultsmentioning

confidence: 99%

Pronunciation Scoring With Goodness of Pronunciation and Dynamic Time Warping

Sheoran¹,

Bajgoti²,

Gupta³

et al. 2023

IEEE Access

View full text Add to dashboard Cite

The current pronunciation scoring based on Goodness of Pronunciation (GOP) uses posterior probabilities of the Acoustic Models. Such algorithms suffer from generalization since they are utilized to determine a score metric for each phoneme rather than on the completeness or comparison with the ideal utterance of the words. In this paper, a novel method is proposed for computing scores calculated using combined scores of prosodic, fluency, completeness, and accuracy. This is achieved using contextaware GOP in conjugation with dynamic time warping (DTW) matching of the pitch contours of a weighted average of the context tokens found in the audio file that is rich in mispronounced phonemes. The proposed work gives flexibility in tuning the results according to different speech aspects based on a single hyperparameter. The results achieved are encouraging and have been validated on the speechocean762 dataset, where Automatic Speech Recognition (ASR) model has been trained on the Librispeech dataset. The resultant mean error of the proposed approach is 3.38% and the value of the correlation coefficient achieved is 0.652.

show abstract

“…Jiaotong Shi et al proposed the contextdependent CaGOP algorithm, which predicts the duration of each phoneme by feeding the reference text into a selfattentive text-based encoder during GOP calculation, and uses the difference between the expected duration and the actual duration of the phoneme obtained by forced alignment as the penalty factor in the GOP calculation [12]. In addition to the studies on spoken language evaluation tasks using GOP algorithms, there are also studies on verbal language evaluation conducted without GOP algorithms, such as wav2vec2.0-based [13], [14], [15] and deep learning featurebased methods [16], but because of the limited speech data available to L2 speakers, training with such methods usually requires the use of pre-trained models and transfer learning. Oral language assessment studies are usually divided into automatic mispronunciation detection and automatic pronunciation quality assessment according to the task's objectives.…”

Section: Related Workmentioning

confidence: 99%

A Multifaceted Approach to Oral Assessment Based on the Conformer Architecture

Fan

Wumaier

et al. 2023

IEEE Access

View full text Add to dashboard Cite

Automatic speaking assessment methods are essential for helping non-native learners to learn native pronunciation. The automated speaking assessment method consists of mispronunciation detection and pronunciation quality assessment. In the past, researchers have usually focused their research on only one specific aspect of the speaking assessment task. Research on multifaceted speaking tasks has been rare, and model building has often led to reduced performance due to the omission of local feature details. In this paper, we propose a multi-width band (MB) method and apply it to the Conformer model. This method can effectively increase the ability of the model to obtain local feature information at different scales. At the same time, we used a multi-task learning approach to train a multifaceted speaking assessment model based on GOP features. We conducted experiments on a self-built monosyllabic Mandarin mispronunciation detection dataset (PSC-MonoSyllable) and an English open-source pronunciation quality assessment dataset (SpeechOcean762), respectively. The experimental results show that the method's mispronunciation detection metrics in terms of phonemes, tones, and words on the PSC-MonoSyllable dataset (F1 scores) reached 70.18%, 80.06%, and 79.82%, respectively. The results of the method on the SpeechOcean 762 dataset for the pronunciation quality assessment task also showed a certain degree of improvement in all aspects of the phoneme-and grapheme-level correlation metrics compared with the baseline model.

show abstract

Automatic Pronunciation Assessment using Self-Supervised Speech Representation Learning

Cited by 5 publications

References 29 publications

An ASR-free Fluency Scoring Approach with Self-Supervised Learning

An ASR-free Fluency Scoring Approach with Self-Supervised Learning

Pronunciation Scoring With Goodness of Pronunciation and Dynamic Time Warping

A Multifaceted Approach to Oral Assessment Based on the Conformer Architecture

Contact Info

Product

Resources

About