Content Modeling for Automated Oral Proficiency Scoring System

Abstract

We developed an automated oral proficiency scoring system for non-native English speakers' spontaneous speech. Automated systems that score holistic proficiency are expected to assess a wide range of performance categories, and the content is one of the core performance categories. In order to assess the quality of the content, we trained a Siamese convolutional neural network (Siamese CNN) to model the semantic relationship between key points generated by experts and a test response. The correlation between human scores and Siamese CNN scores was comparable to human-human agreement (r = 0.63), and it was higher than the baseline content features. The inclusion of Siamese CNN-based feature to the existing state-of-the-art automated scoring model achieved a small but statistically significant improvement. However, the new model suffered from score inflation for long atypical responses with serious content issues. We investigated the reasons of this score inflation by analyzing the associations with linguistic features and identifying areas strongly associated with the score errors.

Automated Speech Scoring System Under The Lens: Evaluating and interpreting the linguistic cues for language proficiency

“…Craighead et al (2020) explored text-based auxiliary tasks and train models in a multi-task manner using speech transcription and found the L1 prediction task to benefit the scoring performance. Recent work also explored specific aspects of speech scoring like response content scoring, where, the features from the transcription of response are modeled with a respective question to learn the relevance of response (Yoon and Lee 2019;Qian et al 2018). Qian et al (2019) build over the work done by Qian et al (2018) and model acoustic cues, prompt, and grammar features to improve scoring performance.…”

Automated Assessment of Task Completion in Spontaneous Speech for Finnish and Finland Swedish Language Learners

“…The research in automated scoring of non-native English speech has shown that it is possible to automatically evaluate the content relevance of a response (Yoon and Lee, 2019). It was demonstrated that fine-tuning Transformer-based models is especially beneficial for this task (Wang et al, 2020).…”

User-centered & Robust NLP OSS: Lessons Learned from Developing & Maintaining RSMTool

“…Each of these groups had separate requirements in terms of entry points, inputs, outputs, and documentation. Only by addressing all of these diverse requirements were we able to achieve wider adoption of RSMTool for model evaluation (Rupp et al, 2019;Yoon and Lee, 2019;Kwong et al, 2020).…”