Examining the Accuracy of a Conversation‐Based Assessment in Interpreting English Learners' Written Responses

Abstract

Substantial progress has been made toward applying technology enhanced conversation‐based assessments (CBAs) to measure the English‐language proficiency of English learners (ELs). CBAs are conversation‐based systems that use conversations among computer‐animated agents and a test taker. We expanded the design and capability of prior conversation‐based instructional and assessment systems and developed a CBA designed to measure the English language skills and the mathematics knowledge of middle school ELs. The prototype CBA simulates an authentic and engaging mathematics classroom where the test taker interacts with two virtual agents to solve math problems. We embedded feedback and supports that are triggered by how the CBA interprets students' written responses. In this study, we administered the CBA to middle school ELs (N = 82) residing in the United States. We examined the extent to which the CBA system was able to consistently interpret the students' responses (722 responses for the 82 students). The study findings helped us to understand the factors that affect the accuracy of the CBA system's interpretations and shed light on how to improve CBA systems that incorporate scaffolding.

Conversation-Based Assessments in Education: Design, Implementation, and Cognitive Walkthroughs for Usability Testing

“…Performance of CBA in Answering Student Responses. Following prior research on the use of CBA for administering different test formats (e.g., Lopez et al, 2021;Ruan et al, 2019), we used two constructed and three selected-response tests. Selected-response tests combined assessment and feedback to measure student knowledge and provide timely feedback.…”

“…They instead tended to guide students using expectation and misconception-tailored (EMT) dialogue. In EMT dialogue, human tutors typically have a list of anticipated correct answers (called expectations) and a list of anticipated incorrect answers (called misconceptions) associated with each question or problem (Graesser et al, 2005; Lopez et al, 2021). They ask questions that are within the student's ability to answer correctly, compare student input with their anticipated expectations and misconceptions, and then provide support when students give an incorrect answer.…”

“…More difficult activities or decisions may require more human participation. CBA is not yet well developed to comprehend content at the level we consider necessary and grade the content quality of student answers (e.g., Lopez et al, 2021). Thus, we must be cognizant of this technology's current limitations.…”

“…The second element (i.e., subject) refers to the topic or area of study that is being supported (e.g., mathematics, history, or English language). For example, AutoTutor worked better when the content was for qualitative domains (Graesser et al, 2005), whereas ELLA-Math was better at reading responses to questions that needed students to write a number than questions that required students to write words (Lopez et al, 2021). However, many of the existing conversational agents have been designed to support the learning of verbal or qualitative content (e.g., Azevedo et al, 2009;Howard et al, 2017;Katz et al, 2021;Rus et al, 2015;Yang & Zapata-Rivera, 2010), rather than numerical or quantitative content (e.g., Lopez et al, 2021).…”

“…Similar to intelligent tutoring systems, CBA relies on the interaction between a student, or more broadly, a learner, and a digital device (e.g., a computer) to administer interactive assessments. The main idea behind CBA is to build a digital assessment environment that measures and supports student learning through interactive conversations generated by a conversational agent (e.g., Lopez et al, 2021). That is, CBA aims to mimic how students would experience a typical conversation on a particular topic and leverage content through interaction to target specific information (Jackson & Zapata-Rivera, 2015).…”