Computerized summary scoring: crowdsourcing-based latent semantic analysis

Abstract

In this study we developed and evaluated a crowdsourcing-based latent semantic analysis (LSA) approach to computerized summary scoring (CSS). LSA is a frequently used mathematical component in CSS, where LSA similarity represents the extent to which the to-be-graded target summary is similar to a model summary or a set of exemplar summaries. Researchers have proposed different formulations of the model summary in previous studies, such as pregraded summaries, expert-generated summaries, or source texts. The former two methods, however, require substantial human time, effort, and costs in order to either grade or generate summaries. Using source texts does not require human effort, but it also does not predict human summary scores well. With human summary scores as the gold standard, in this study we evaluated the crowdsourcing LSA method by comparing it with seven other LSA methods that used sets of summaries from different sources (either experts or crowdsourced) of differing quality, along with source texts. Results showed that crowdsourcing LSA predicted human summary scores as well as expert-good and crowdsourcing-good summaries, and better than the other methods. A series of analyses with different numbers of crowdsourcing summaries demonstrated that the number (from 10 to 100) did not significantly affect performance. These findings imply that crowdsourcing LSA is a promising approach to CSS, because it saves human effort in generating the model summary while still yielding comparable performance. This approach to small-scale CSS provides a practical solution for instructors in courses, and also advances research on automated assessments in which student responses are expected to semantically converge on subject matter content.

A design experiment on technology‐based learning progress feedback in a graduate‐level online course

“…Students in this study endorsed the guiding messages we provided (i.e., simple and generic statements) as a supportive tool. Irrespective of message simplicity, students perceived that such elaboration, beyond mere verification (e.g., scores or levels in the legend), was useful, as found in studies about the advantages of elaborate feedback in formative feedback technologies (e.g., Berlanga et al, 2012; H. Li et al, 2018; Rakoczy et al, 2013; Sedrakyan et al, 2019; Sung et al, 2016). Also, those messages can work as motivational and emotional factors in LPF design that likely promote students' mastery goals (Britner & Pajares, 2006; Nokes‐Malach & Mestre, 2013).…”

Improving summary writing through formative feedback in a technology‐enhanced learning environment

“…The SMART assessment centres on the extent to which learners internalized the information from reading material. Although the 3S indices of mental models reflect the text structure (e.g., how cohesively a student writes a summary) and lexical diversity of a summary (Crossley, Kyle, Davenport, & McNamara, 2016; Li et al, 2018), SMART's feedback tends to focus on content coverage in terms of key ideas and their relations. The current SMART approach is beneficial for “writing to learn,” rather than “learning to write.” Further work could explore the effects of embedding summary writing instruction (Friend, 2001; Stevens et al, 2019) and composition feedback (Wade‐Stein & Kintsch, 2004) in addition to SMART'S existing content‐driven feedback.…”

“…Automated summary evaluators (ASEs) leverage advanced natural language processing tools and techniques to assess linguistic features of students' written responses (Allen, Jacovina, & McNamara, 2016; Passonneau et al, 2018; Strobl et al, 2019). ASEs such as Summary Street (Wade‐Stein & Kintsch, 2004), Online Summary Assessment and Feedback System (Sung, Liao, Chang, Chen, & Chang, 2016), crowd‐source summary evaluation (Li, Cai, & Graesser, 2016, 2018), ROUGE (Lin, 2004), SEMILAR (Rus, D'Mello, Hu, & Graesser, 2013) and PryEval (Gao, Warner, & Passonneau, 2019) use hundreds of descriptive linguistic indices related to word‐level (e.g., lexical diversity), sentence‐level (e.g., syntactic complexity) and document‐level (e.g., cohesion from sentence to sentence) qualities of the summary to examine the quality of writing and to drive feedback to help students improve their summary writing skills. However, feedback in ASEs tend to focus on the act of summarizing, rather than comprehension (Sung et al, 2016).…”

Scoring Summaries Using Recurrent Neural Networks

“…Others have used distribution-similarity measures such as Kullback-Leibler (KL) divergence and Jensen Shannon (JS) divergence [21,43], textual entailment [44] and crowdsourcing based LSA [18] for evaluating summaries. However, relatively few studies have used machinelearning techniques for summary evaluation beyond the aforementioned regressionbased approaches [45][46][47].…”

“…Although summarization practice has proven effectiveness, teachers can find it challenging to implement practice activities because evaluating student summaries requires a great deal of effort and time [18]. Automated methods for summary evaluation traditionally involve evaluating quality metrics such as readability, content, conciseness, coherence and grammar [19].…”