Is a Wizard-of-Oz Required for Robot-Led Conversation Practice in a Second Language?

Abstract

The large majority of previous work on human-robot conversations in a second language has been performed with a human wizard-of-Oz. The reasons are that automatic speech recognition of non-native conversational speech is considered to be unreliable and that the dialogue management task of selecting robot utterances that are adequate at a given turn is complex in social conversations. This study therefore investigates if robot-led conversation practice in a second language with pairs of adult learners could potentially be managed by an autonomous robot. We first investigate how correct and understandable transcriptions of second language learner utterances are when made by a state-of-the-art speech recogniser. We find both a relatively high word error rate (41%) and that a substantial share (42%) of the utterances are judged to be incomprehensible or only partially understandable by a human reader. We then evaluate how adequate the robot utterance selection is, when performed manually based on the speech recognition transcriptions or autonomously using (a) predefined sequences of robot utterances, (b) a general state-of-the-art language model that selects utterances based on learner input or the preceding robot utterance, or (c) a custom-made statistical method that is trained on observations of the wizard’s choices in previous conversations. It is shown that adequate or at least acceptable robot utterances are selected by the human wizard in most cases (96%), even though the ASR transcriptions have a high word error rate. Further, the custom-made statistical method performs as well as manual selection of robot utterances based on ASR transcriptions. It was also found that the interaction strategy that the robot employed, which differed regarding how much the robot maintained the initiative in the conversation and if the focus of the conversation was on the robot or the learners, had marginal effects on the word error rate and understandability of the transcriptions but larger effects on the adequateness of the utterance selection. Autonomous robot-led conversations may hence work better with some robot interaction strategies.

Adaptive Robot Discourse for Language Acquisition in Adulthood

“…Our findings corroborated that ASR performance for non-native speakers is worse than for L1 speakers, increasing up to a double of the Word Error Rate (WER) of native speakers in social conversations. Our recent work further checked how these results may alter the performance of the robot's autonomous dialogue system, finding that proposed automatic utterance selection methods performed adequately for different dialogue strategies [20]. We further plan to adapt the language model of the best-performed ASR to improve speech recognition.…”

Text Recognition for Library Collection in Different Light Conditions

“…Meanwhile, CER and WER have been applied to detect errors and accuracy in children's writing [20]. However, in its implementation, it is often utilized in speech recognition because its use can detect the accuracy of many sentences [21], [22].…”

A Systematic Literature Review on the Applications of Robots and Natural Language Processing in Education

“…13 Engwal et al [73] Assess the feasibility of autonomous robot-led conversations for second language practice and evaluate speech recognition and utterance selection methods…”