Bootstrapping Development of a Cloud‐Based Spoken Dialog System in the Educational Domain From Scratch Using Crowdsourced Data

Ramanarayanan, Vikram; Suendermann-Oeft, David; Lange, Patrick; Ivanov, Alexei V.; Evanini, Keelan; Yu, Zhou; Tsuprun, Eugene; Ye, Qian

doi:10.1002/ets2.12105

Cited by 13 publications

(19 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To test the basic functionality of the presented multimodal dialog system, we are presently conducting a user study deploying multiple language assessment prototypes in a crowdsourcing environment. As of September 2016, more than 20,000 calls have been processed by the system (see a preliminary analysis in Ramanarayanan et al, ).…”

Section: Discussionmentioning

confidence: 99%

A Multimodal Dialog System for Language Assessment: Current State and Future Directions

Suendermann-Oeft

Ramanarayanan

et al. 2017

ETS Research Report Series

Self Cite

View full text Add to dashboard Cite

We present work in progress on a multimodal dialog system for English language assessment using a modular cloud‐based architecture adhering to open industry standards. Among the modules being developed for the system, multiple modules heavily exploit machine learning techniques, including speech recognition, spoken language proficiency rating, speaker recognition, and the scoring of behaviors in multimodal data streams.

show abstract

Section: Discussionmentioning

confidence: 99%

A Multimodal Dialog System for Language Assessment: Current State and Future Directions

Suendermann-Oeft

Ramanarayanan

et al. 2017

ETS Research Report Series

Self Cite

View full text Add to dashboard Cite

show abstract

“…There is an increasing demand for dialog-based learning and assessment solutions at scale, given the rapidly growing language learning and online education marketplace. Dialog system technologies are one solution capable of addressing and automating this demand at scale (Ramanarayanan et al, 2016). However, such conversational technologies need to be able to provide useful and actionable feedback to users in order for them to be widely adopted.…”

Section: Automated Scoring Of Text Dialogmentioning

confidence: 99%

Exploring Recurrent, Memory and Attention Based Architectures for Scoring Interactional Aspects of Human-Machine Text Dialog

Ramanarayanan,

Mulholland,

Ghosh

2020

Preprint

Self Cite

View full text Add to dashboard Cite

An important step towards enabling English language learners to improve their conversational speaking proficiency involves automated scoring of multiple aspects of interactional competence and subsequent targeted feedback. This paper builds on previous work in this direction to investigate multiple neural architectures -recurrent, attention and memory based -along with feature-engineered models for the automated scoring of interactional and topic development aspects of text dialog data. We conducted experiments on a conversational database of text dialogs from human learners interacting with a cloud-based dialog system, which were triple-scored along multiple dimensions of conversational proficiency. We find that fusion of multiple architectures performs competently on our automated scoring task relative to expert interrater agreements, with (i) hand-engineered features passed to a support vector learner and (ii) transformer-based architectures contributing most prominently to the fusion.

show abstract

“…We used Amazon Mechanical Turk for our crowdsourcing data collection experiments. Crowdsourcing has been used in the past for the assessment of dialog systems as well as for collection of dialog interactions (see for instance (McGraw et al, 2010;Rayner et al, 2011;Jurcıcek et al, 2011;Ramanarayanan et al, 2016)). In addition to interacting with the text chatbot interface to complete the conversational task, workers were requested to fill out a 2-3 minute survey regarding different aspects of the interaction, such as their overall experience, how engaged they felt while interacting with the system, how well the system understood them, and basic demographic information.…”

Section: Crowdsourcing Data Collectionmentioning

confidence: 99%

Toward Automatically Measuring Learner Ability from Human-Machine Dialog Interactions using Novel Psychometric Models

Ramanarayanan¹,

LaMar

2018

Proceedings of the Thirteenth Workshop on Innovative Use of NLP For Building Educational Applications

Self Cite

View full text Add to dashboard Cite

While dialog systems have been widely deployed for computer-assisted language learning (CALL) and formative assessment systems in recent years, relatively limited work has been done with respect to the psychometrics and validity of these technologies in evaluating and providing feedback regarding student learning and conversational ability. This paper formulates a Markov decision process based measurement model, and applies it to text chat data collected from crowdsourced native and non-native English language speakers interacting with an automated dialog agent. We investigate how well the model measures speaker conversational ability, and find that it effectively captures the differences in how native and non-native speakers of English accomplish the dialog task. Such models could have important implications for CALL systems of the future that effectively combine dialog management with measurement of learner conversational ability in real-time.

show abstract

Bootstrapping Development of a Cloud‐Based Spoken Dialog System in the Educational Domain From Scratch Using Crowdsourced Data

Cited by 13 publications

References 12 publications

A Multimodal Dialog System for Language Assessment: Current State and Future Directions

A Multimodal Dialog System for Language Assessment: Current State and Future Directions

Exploring Recurrent, Memory and Attention Based Architectures for Scoring Interactional Aspects of Human-Machine Text Dialog

Toward Automatically Measuring Learner Ability from Human-Machine Dialog Interactions using Novel Psychometric Models

Contact Info

Product

Resources

About