Deep neural network acoustic models for spoken assessment applications

Cheng, Jian; Chen, Xin; Metallinou, Angeliki

doi:10.1016/j.specom.2015.07.006

Cited by 43 publications

(30 citation statements)

References 62 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Pearson embraced new high-accuracy DNN ASR technology in its speech scoring research with significantly increased recognition accuracy on nonnative speech. More details on Pearson's work utilizing a DNN ASR can be found in Cheng, Chen, and Metallinou (2015).…”

Section: Randd Activitiesmentioning

confidence: 99%

Automated Scoring of Nonnative Speech Using the SpeechRater^SMv. 5.0 Engine

Chen

Zechner

Yoon

et al. 2018

ETS Research Report Series

View full text Add to dashboard Cite

This research report provides an overview of the R&D efforts at Educational Testing Service related to its capability for automated scoring of nonnative spontaneous speech with the SpeechRaterSM automated scoring service since its initial version was deployed in 2006. While most aspects of this R&D work have been published in various venues in recent years, no comprehensive account of the current state of SpeechRater has been provided since the initial publications following its first operational use in 2006. After a brief review of recent related work by other institutions, we summarize the main features and feature classes that have been developed and introduced into SpeechRater in the past 10 years, including features measuring aspects of pronunciation, prosody, vocabulary, grammar, content, and discourse. Furthermore, new types of filtering models for flagging nonscorable spoken responses are described, as is our new hybrid way of building linear regression scoring models with improved feature selection. Finally, empirical results for SpeechRater 5.0 (operationally deployed in 2016) are provided.

show abstract

Section: Randd Activitiesmentioning

confidence: 99%

Automated Scoring of Nonnative Speech Using the SpeechRater^SMv. 5.0 Engine

Chen

Zechner

Yoon

et al. 2018

ETS Research Report Series

View full text Add to dashboard Cite

show abstract

“…Moreover, the multi-column deepstacking DNN based ensemble classifier was better when the heterogeneous features had significant performance differences. In addition, we expected that the more detailed experiments of the DNN configurations and the feature combinations would improve the performance, especially using a more large amount data (Cheng et al, 2015).…”

Section: Dnn-based Ensemble Classifiermentioning

confidence: 99%

A deep-learning based native-language classification by using a latent semantic analysis for the NLI Shared Task 2017

Jeon

Song

et al. 2017

Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications

View full text Add to dashboard Cite

This paper proposes a deep-learning based native-language identification (NLI) using a latent semantic analysis (LSA) as a participant (ETRI-SLP) of the NLI Shared Task 2017 where the NLI Shared Task 2017 aims to detect the native language of an essay or speech response of a standardized assessment of English proficiency for academic purposes. To this end, we use the six unit forms of a text data such as character 4/5/6-grams and word 1/2/3-grams. For each unit form of text data, we convert it into a count-based vector, extract a 2000-rank LSA feature, and perform a linear discriminant analysis (LDA) based dimension reduction. From the count-based vector or the LSA-LDA feature, we also obtain the output prediction values of a support vector machine (SVM) based classifier, the output prediction values of a deep neural network (DNN) based classifier, and the bottleneck values of a DNN based classifier. In order to incorporate the various kinds of text-based features and a speech-based i-vector feature, we design two DNN based ensemble classifiers for late fusion and early fusion, respectively. From the NLI experiments, the F1 (macro) scores are obtained as 0.8601, 0.8664, and 0.9220 for the essay track, the speech track, and the fusion track, respectively. The proposed method has comparable performance to the top-ranked teams for the speech and fusion tracks, although it has slightly lower performance for the essay track.

show abstract

“…Most approaches require the distractor and the target word to have the same part-of-speech (POS) and similar level of difficulty, often approximated by word frequency (Coniam, 1997;Shei, 2001;. They must also be semantically close, which can be quantified with semantic distance in WordNet (Lin et al, 2007;Pino et al, 2008;Chen et al, 2015;Susanti et al, 2015), thesauri (Sumita et al, 2005;Smith et al, 2010), ontologies (Karamanis et al, 2006;Ding and Gu, 2010), or handcrafted rules (Chen et al, 2006). Another approach generates distractors that are semantically similar to the target word in some sense, but not in the particular sense in the carrier sentence (Zesch and Melamud, 2014).…”

Section: Previous Workmentioning

confidence: 99%

Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications

Tetreault

Burstein

Leacock³

et al. 2017

View full text Add to dashboard Cite

Since the first workshop in 1997, BEA has become the leading venue for sharing and publishing innovative work that uses NLP to develop educational applications. The consistent interest and growth of the workshop has clear ties to challenges in education. The research presented at the workshop highlights advances in the technology and the maturity of the field of NLP in education. The capabilities serve as a response to educational challenges and are poised to support the needs of a variety of stakeholders, including educators, learners, parents, and administrators.NLP capabilities now support an array of learning domains, including writing, speaking, reading, and mathematics. In the writing and speech domains, automated writing evaluation (AWE) and speech assessment applications, respectively, are commercially deployed in high-stakes assessment and instructional settings, including Massive Open Online Courses (MOOCs). We also see widelyused commercial applications for plagiarism detection and peer review and explosive growth of mobile applications for game-based applications for instruction and assessment. The current educational and assessment landscape continues to foster a strong interest and high demand that pushes the state of the art in AWE capabilities to expand the analysis of written responses to writing genres other than those traditionally found in standardized assessments, especially writing tasks requiring use of sources and argumentative discourse.Steady growth in the development of NLP-based applications for education has prompted an increased number of workshops that typically focus on a single subfield. In BEA, we make an effort to have papers from many subfields, for example, tools for automated scoring, automated test-item generation, curriculum development, evaluation of text, dialogue, evaluation of genres beyond essays, feedback studies, and grammatical error correction.This year we received a record 62 submissions, and accepted 9 papers as oral presentations and 25 as poster presentation and/or demos, for an overall acceptance rate of 55 percent. Each paper was reviewed by three members of the Program Committee who were believed to be most appropriate for each paper. We continue to have a very strong policy to deal with conflicts of interest. First, we made a concerted effort to not assign papers to reviewers to evaluate if the paper had an author from their institution. Second, with respect to the organizing committee, authors of papers for which there was a conflict of interest recused themselves from the discussions.While the field is growing, we do recognize that there is a core group of institutions and researchers who work in this area. With a higher acceptance rate, we were able to include papers from a wider variety of topics and institutions. The papers accepted were selected on the basis of several factors, including the relevance to a core educational problem space, the novelty of the approach or domain, and the strength of the research. The accepted papers were highly diverse -...

show abstract

Deep neural network acoustic models for spoken assessment applications

Cited by 43 publications

References 62 publications

Automated Scoring of Nonnative Speech Using the SpeechRater^SMv. 5.0 Engine

Automated Scoring of Nonnative Speech Using the SpeechRater^SMv. 5.0 Engine

A deep-learning based native-language classification by using a latent semantic analysis for the NLI Shared Task 2017

Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications

Contact Info

Product

Resources

About

Deep neural network acoustic models for spoken assessment applications

Cited by 43 publications

References 62 publications

Automated Scoring of Nonnative Speech Using the SpeechRaterSMv. 5.0 Engine

Automated Scoring of Nonnative Speech Using the SpeechRaterSMv. 5.0 Engine

A deep-learning based native-language classification by using a latent semantic analysis for the NLI Shared Task 2017

Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications

Contact Info

Product

Resources

About

Automated Scoring of Nonnative Speech Using the SpeechRater^SMv. 5.0 Engine

Automated Scoring of Nonnative Speech Using the SpeechRater^SMv. 5.0 Engine