Speech Processing for Language Learning: A Practical Approach to Computer-Assisted Pronunciation Teaching

Abstract

This article contributes to the discourse on how contemporary computer and information technology may help in improving foreign language learning not only by supporting better and more flexible workflow and digitizing study materials but also through creating completely new use cases made possible by technological improvements in signal processing algorithms. We discuss an approach and propose a holistic solution to teaching the phonological phenomena which are crucial for correct pronunciation, such as the phonemes; the energy and duration of syllables and pauses, which construct the phrasal rhythm; and the tone movement within an utterance, i.e., the phrasal intonation. The working prototype of StudyIntonation Computer-Assisted Pronunciation Training (CAPT) system is a tool for mobile devices, which offers a set of tasks based on a “listen and repeat” approach and gives the audio-visual feedback in real time. The present work summarizes the efforts taken to enrich the current version of this CAPT tool with two new functions: the phonetic transcription and rhythmic patterns of model and learner speech. Both are designed on a base of a third-party automatic speech recognition (ASR) library Kaldi, which was incorporated inside StudyIntonation signal processing software core. We also examine the scope of automatic speech recognition applicability within the CAPT system workflow and evaluate the Levenstein distance between the transcription made by human experts and that obtained automatically in our code. We developed an algorithm of rhythm reconstruction using acoustic and language ASR models. It is also shown that even having sufficiently correct production of phonemes, the learners do not produce a correct phrasal rhythm and intonation, and therefore, the joint training of sounds, rhythm and intonation within a single learning environment is beneficial. To mitigate the recording imperfections voice activity detection (VAD) is applied to all the speech records processed. The try-outs showed that StudyIntonation can create transcriptions and process rhythmic patterns, but some specific problems with connected speech transcription were detected. The learners feedback in the sense of pronunciation assessment was also updated and a conventional mechanism based on dynamic time warping (DTW) was combined with cross-recurrence quantification analysis (CRQA) approach, which resulted in a better discriminating ability. The CRQA metrics combined with those of DTW were shown to add to the accuracy of learner performance estimation. The major implications for computer-assisted English pronunciation teaching are discussed.

An Open CAPT System for Prosody Practice: Practical Steps towards Multilingual Setup

“…Repurposed tools Forvo (Pierson 2015), Lingua Libre (https://lingualibre.org/) accessed on 22 December 2023, YouGlish (McCarthy 2018) 3 Dedicated tools CAPT system for hearing impaired (Sztahó et al 2014), Inton@Trainer (Lobanov et al 2018), ELSA Speak (Samad and Ismail 2020), StudyIntonation (Boitsova et al 2018;Bogach et al 2021) Thus, in our work we strive to rise to the challenges of the personalization and customization of CAPT feedback to both the target language and the individual learners. Although most CAPT systems address a single language, our aim is to develop an environment suitable for all languages, allowing the visualizations to be tailored to the prosody of the target language.…”

“…Estimating and measuring fundamental frequency and modeling the pitch produced are non-trivial tasks (Hirst and de Looze 2021), which may require an external reference to verify all of the stages. It was shown in Bogach et al (2021) how Praat software could be of much assistance when live speech recording for pitch extraction is conducted; voice activity detection algorithms are used on the raw pitch readings before the other signal conditioning stages. The latter stages include pitch filtering, approximation, and smoothing, which are standard digital signal processing stages to obtain a pitch curve adapted for pedagogic purposes.…”

Automatic speech recognition and text-to-speech technologies for L2 pronunciation improvement

“…ASR technology applied to a CALL context has been criticized by some authors in the past decade due to its incapacity to comprehend L2 speech accurately at a similar rate as human listeners (DERWING; MUNRO; CARBONARO, 2000;KIM, 2006;LEVIS;SUVOROV, 2013), its much lower accuracy scores for nonnative speakers than those for native speakers (ASHWELL; ELAM, 2017; ROGERSON-REVELL, 2021), and its insufficient or even incorrect feedback (CHEN, 2011;DE-MENKO;WAGNER;CYLWIK, 2010;LEVIS;SUVOROV, 2013;ROGERSON-REVELL, 2021). On the other hand, recent research has shown that this technology has been improving in the past years (ASHWELL; ELAM, 2017;DIZON, 2020;DIZON;TANG, 2020;MCCROCKLIN;EDALATISHAMS, 2020;MOUSSALLI;CARDOSO, 2020;BOGACH et al, 2021). Furthermore, Ashwell and Elam (2017, p. 61) argue that "these systems are continually improving on their respective accuracy rates by constantly gathering acoustic information and utilizing machine learning".…”

A study of college students’ perceptions of utilizing automatic speech recognition technology to assist English oral proficiency

“…Some ASR technologies provide feedback in the form of verbal descriptions and articulation charts. This feedback can improve learners’ pronunciation gradually and enhance the accuracy of segmentation ( Bogach et al, 2021 ).…”