3M: An Effective Multi-view, Multi-granularity, and Multi-aspect Modeling Approach to English Pronunciation Assessment

Chao, Fu-An; Lo, Tien-Hong; Wu, Ting-Jung; Sung, Yao–Ting; Chen, Berlin

doi:10.23919/apsipaasc55919.2022.9979979

Cited by 12 publications

(15 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Table 4 compares the proposed approach with various other approaches using different datasets such as Spee-chocean762, TIMIT, LibriSpeech, and more. Latest models such as HuBERT [35] and Wav2Vec2 [36] were also compared. It should be noted that our model was not fine-tuned whereas all other models were fine-tuned before getting these results.…”

Section: Resultsmentioning

confidence: 99%

Pronunciation Scoring With Goodness of Pronunciation and Dynamic Time Warping

Sheoran¹,

Bajgoti²,

Gupta³

et al. 2023

IEEE Access

View full text Add to dashboard Cite

The current pronunciation scoring based on Goodness of Pronunciation (GOP) uses posterior probabilities of the Acoustic Models. Such algorithms suffer from generalization since they are utilized to determine a score metric for each phoneme rather than on the completeness or comparison with the ideal utterance of the words. In this paper, a novel method is proposed for computing scores calculated using combined scores of prosodic, fluency, completeness, and accuracy. This is achieved using contextaware GOP in conjugation with dynamic time warping (DTW) matching of the pitch contours of a weighted average of the context tokens found in the audio file that is rich in mispronounced phonemes. The proposed work gives flexibility in tuning the results according to different speech aspects based on a single hyperparameter. The results achieved are encouraging and have been validated on the speechocean762 dataset, where Automatic Speech Recognition (ASR) model has been trained on the Librispeech dataset. The resultant mean error of the proposed approach is 3.38% and the value of the correlation coefficient achieved is 0.652.

show abstract

Section: Resultsmentioning

confidence: 99%

Pronunciation Scoring With Goodness of Pronunciation and Dynamic Time Warping

Sheoran¹,

Bajgoti²,

Gupta³

et al. 2023

IEEE Access

View full text Add to dashboard Cite

show abstract

“…All this emphasizes that it has a high potential to be used to obtain important characteristics. Among the successes reported, one of them is encouraging results in the assessment of English pronunciation [14]. Likewise with the successful use of prosody features in identifying the correct recitation of the Qur'an [15].…”

Section: Related Workmentioning

confidence: 93%

Dynamic Time Warping Features Extraction Design for Quranic Syllable-based Harakaat Assessment

Shafie¹,

Azizan²,

Adam³

et al. 2022

IJACSA

View full text Add to dashboard Cite

The use of technological speech recognition systems with a variety of approaches and techniques has grown exponentially in varieties of human-machine interaction applications. The assessment for Qur'anic recitation errors based on syllables utterance is used to meet the Tajweed rules which generally consist of Harakaat (prolonging). The digital transformation of Quranic voice signals with identification of Tajweed-based recitation errors of Harakaat is the main research work in this paper. The study focused on speech processing implemented using the representation of Quranic Recitation Speech Signals (QRSS) in the best digital format based on Al-Quran syllables and feature extraction design to reveal similarities or differences in recitation (based on Al-Quran syllables) between experts and student. The method of Dynamic Time Warping (DTW) is used as Short Time Frequency Transform (STFT) of QRSS syllable feature for Harakaat measurement. Findings from this paper include an approach based on human-guidance threshold classification that is used specifically to evaluate Harakaat based on the syllables of the Qur'an. The threshold classification performance obtained for Harakaat is above 80% in the training and testing stages. The results of the analysis at the end of the experiment have concluded that the threshold classification method for Minimum Path Cost (MPC) feature parameters can be used as an important feature to evaluate the rules of Tajwid Harakaat embedded in syllables.

show abstract

“…A follow-up work attempted to further improve the system by using multi-view representations that * This work was done during an internship at ByteDance AI Lab. comprise additional prosodic and SSL speech features [15] . In [16], Bi-directional Long Short Term Memory (BLSTM) is adopted to predict fluency scores from a sequence of phone-level deep features (usually extracted from the bottleneck layer of a DNN acoustic model) and its corresponding phone duration features.…”

Section: Introductionmentioning

confidence: 99%

An ASR-free Fluency Scoring Approach with Self-Supervised Learning

Liu¹,

Fu²,

Tian³

et al. 2023

Preprint

View full text Add to dashboard Cite

A typical fluency scoring system generally relies on an automatic speech recognition (ASR) system to obtain time stamps in input speech for either the subsequent calculation of fluency-related features or directly modeling speech fluency with an end-to-end approach. This paper describes a novel ASR-free approach for automatic fluency assessment using self-supervised learning (SSL). Specifically, wav2vec2.0 is used to extract frame-level speech features, followed by K-means clustering to assign a pseudo label (cluster index) to each frame. A BLSTM-based model is trained to predict an utterance-level fluency score from frame-level SSL features and the corresponding cluster indexes. Neither speech transcription nor time stamp information is required in the proposed system. It is ASR-free and can potentially avoid the ASR errors effect in practice. Experimental results carried out on non-native English databases show that the proposed approach significantly improves the performance in the "open response" scenario as compared to previous methods and matches the recently reported performance in the "read aloud" scenario.

show abstract

3M: An Effective Multi-view, Multi-granularity, and Multi-aspect Modeling Approach to English Pronunciation Assessment

Cited by 12 publications

References 29 publications

Pronunciation Scoring With Goodness of Pronunciation and Dynamic Time Warping

Pronunciation Scoring With Goodness of Pronunciation and Dynamic Time Warping

Dynamic Time Warping Features Extraction Design for Quranic Syllable-based Harakaat Assessment

An ASR-free Fluency Scoring Approach with Self-Supervised Learning

Contact Info

Product

Resources

About