Automatic evaluation of spoken summaries: the case of language assessment

Abstract

This paper investigates whether ROUGE, a popular metric for the evaluation of automated written summaries, can be applied to the assessment of spoken summaries produced by non-native speakers of English. We demonstrate that ROUGE, with its emphasis on the recall of information, is particularly suited to the assessment of the summarization quality of non-native speakers' responses. A standard baseline implementation of ROUGE-1 computed over the output of the automated speech recognizer has a Spearman correlation of ρ = 0.55 with experts' scores of speakers' proficiency (ρ = 0.51 for a content-vector baseline). Further increases in agreement with experts' scores can be achieved by using types instead of tokens for the computation of word frequencies for both candidate and reference summaries, as well as by using multiple reference summaries instead of a single one. These modifications increase the correlation with experts' scores to a Spearman correlation of ρ = 0.65. Furthermore, we found that the choice of reference summaries does not have any impact on performance, and that the adjusted metric is also robust to errors introduced by automated speech recognition (ρ = 0.67 for human transcriptions vs. ρ = 0.65 for speech recognition output).

Controlling Length in Abstractive Summarization Using a Convolutional Neural Network

“…We use significance test to prove that similarity metric is reliable even though the numerical difference of similarity scores in experiment is little. Because the similarity scores of generated summaries do not follow normal distribution, we take Kruskal-Wallis test (Loukina et al, 2014;Albert, 2017) as our significance test to measure that the difference of similarity results of three methods is significant or not. As shown in Table 9, all pvalues are less than 0.05.…”

Automated Scoring of Nonnative Speech Using the SpeechRaterSMv. 5.0 Engine

“…In addition to the relatively straightforward method of using CVA models and cosine similarity calculations to produce the content features, additional approaches have been investigated for scoring spontaneous speech. Some of these include using latent semantic analysis (LSA; Metallinou & Cheng, ), pointwise mutual information (Xie, Evanini, & Zechner, ), and the ROUGE summarization evaluation metric (Lin & Rey, ; Loukina, Zechner, & Chen, ).…”

Performance of Automated Speech Scoring on Different Low‐ to Medium‐Entropy Item Types for Low‐Proficiency English Learners

“…Since the early 2000s, several groups have built systems for scoring less constrained and more unpredictable speaking items, which incorporated additional sources of information for scoring, for example, diversity of vocabulary or grammatical complexity (Bernstein et al, ; Chen & Zechner, ; Strik, Van De Loo, Van Doremalen, & Cucchiarini, ; Yoon, Bhat, & Zechner, ; Zechner, Higgins, Xi, & Williamson, ). Recent work has also looked at evaluating the content relevance of spoken responses (Loukina, Zechner, & Chen, ; Somasundaran, Lee, Chodorow, & Wang, ; Xie, Evanini, & Zechner, ).…”