Assessing comprehension during reading with the Reading Strategy Assessment Tool (RSAT)

Abstract

Comprehension emerges as the results of inference and strategic processes that support the construction of a coherent mental model for a text. However, the vast majority of comprehension skills tests adopt a format that does not afford an assessment of these processes as they operate during reading. This study assessed the viability of the Reading Strategy Assessment Tool (RSAT), which is an automated computer-based reading assessment designed to measure readers’ comprehension and spontaneous use of reading strategies while reading texts. In the tool, readers comprehend passages one sentence at a time, and are asked either an indirect (“What are your thoughts regarding your understanding of the sentence in the context of the passage?”) or direct (e.g., why X?) question after reading each pre-selected target sentence. The answers to the indirect questions are analyzed on the extent that they contain words associated with comprehension processes. The answers to direct questions are coded for the number of content words in common with an ideal answer, which is intended to be an assessment of emerging comprehension. In the study, the RSAT approach was shown to predict measures of comprehension comparable to standardized tests. The RSAT variables were also shown to correlate with human ratings. The results of this study constitute a “proof of concept” and demonstrate that it is possible to develop a comprehension skills assessment tool that assesses both comprehension and comprehension strategies.

Computer-based assessment of student-constructed responses

“…The last example illustrates that some students do elaborate beyond the discourse contexts, but these types of responses support comprehension only to the extent that they are relevant to the text. As we will discuss below, detecting and classifying these kinds of elaborative responses is a major challenge because it is difficult to anticipate the variety of responses that students can produce (Magliano et al, 2011;Millis, Magliano, Todaro, & McNamara, 2007).…”

“…Nonetheless, the development of automated assessments is warranted because these protocols expose individual differences in approaches to reading, particularly in the realm of prior domain knowledge and use of comprehension strategies (Chi, Bassok, Lewis, Reimann, & Glaser, 1989;Coté et al, 1998;Magliano & Millis, 2003, Magliano et al, 2011. For example, less skilled readers tend to paraphrase the sentence they just read, whereas skilled readers tend to bridge explicit statements in the text and elaborate the material with text-relevant inferences (Magliano & Millis, 2003;Magliano et al, 2011). Moreover, computer systems that train students to think aloud or self-explain effectively have been shown to promote comprehension .…”

“…If the readers' content has words that match or are close synonyms to the words in the current sentence, the researcher can readily infer that the readers are restating or paraphrasing the sentence. But if the words draw heavily upon content from sentences in the prior discourse, readers are likely to be engaging in the process of generating bridging inferences (Magliano & Millis, 2003;Magliano et al, 2011). Strategy detection systems, such as the Reading Strategy Assessment Tool (RSAT; Magliano et al, 2011), provide reasonable estimates of broad categories of process/ strategy, such as paraphrasing and bridging, in the context of generating think-aloud or self-explanation protocols while text is read.…”

“…But if the words draw heavily upon content from sentences in the prior discourse, readers are likely to be engaging in the process of generating bridging inferences (Magliano & Millis, 2003;Magliano et al, 2011). Strategy detection systems, such as the Reading Strategy Assessment Tool (RSAT; Magliano et al, 2011), provide reasonable estimates of broad categories of process/ strategy, such as paraphrasing and bridging, in the context of generating think-aloud or self-explanation protocols while text is read. This is accomplished by basing assessments of processes on the content of the protocols (e.g., information from current or prior discourse sentences).…”

“…We define student-constructed responses as those that require a student to produce an answer in natural language that may range from a couple of sentences to several paragraphs. These advances have been in the context of computer-based assessments of explanations and think-aloud protocols during reading comprehension (Gilliam, Magliano, Millis, Levinstein, & Boonthum, 2007;Magliano, Millis, the RSAT Development Team, Levinstein, & Boonthum, 2011), the grading of essays and text summaries (Attali & Burstein, 2006;Burstein, Marcu, & Knight, 2003;Franzke, Kintsch, Caccamise, Johnson, & Dooley, 2005;Landauer, Laham, & Foltz, 2003), the grading of short-answer questions (Leacock & Chodorow, 2003), and intelligent tutoring systems and trainers that require students to produce constructed responses during interactive conversations (Graesser, Jeon, & Dufty, 2008;Litman et al, 2006;McNamara, Levinstein, & Boonthum, 2004;VanLehn et al, 2007). These can take the form of directed responses to specific questions or less directed thinkaloud and self-explanation responses.…”