Using automated analysis to assess middle school students' competence with scientific argumentation

Abstract

Argumentation is fundamental to science education, both as a prominent feature of scientific reasoning and as an effective mode of learning—a perspective reflected in contemporary frameworks and standards. The successful implementation of argumentation in school science, however, requires a paradigm shift in science assessment from the measurement of knowledge and understanding to the measurement of performance and knowledge in use. Performance tasks requiring argumentation must capture the many ways students can construct and evaluate arguments in science, yet such tasks are both expensive and resource‐intensive to score. In this study we explore how machine learning text classification techniques can be applied to develop efficient, valid, and accurate constructed‐response measures of students' competency with written scientific argumentation that are aligned with a validated argumentation learning progression. Data come from 933 middle school students in the San Francisco Bay Area and are based on three sets of argumentation items in three different science contexts. The findings demonstrate that we have been able to develop computer scoring models that can achieve substantial to almost perfect agreement between human‐assigned and computer‐predicted scores. Model performance was slightly weaker for harder items targeting higher levels of the learning progression, largely due to the linguistic complexity of these responses and the sparsity of higher‐level responses in the training data set. Comparing the efficacy of different scoring approaches revealed that breaking down students' arguments into multiple components (e.g., the presence of an accurate claim or providing sufficient evidence), developing computer models for each component, and combining scores from these analytic components into a holistic score produced better results than holistic scoring approaches. However, this analytical approach was found to be differentially biased when scoring responses from English learners (EL) students as compared to responses from non‐EL students on some items. Differences in the severity between human and computer scores for EL between these approaches are explored, and potential sources of bias in automated scoring are discussed.

AI and formative assessment: The train has left the station

“…Admittedly, AI cannot provide accurate scores for all students, but neither can teachers. We argue that scores do not necessarily lead to inequity unless the automatic scores are overly inaccurate, which is not the case in our study and most published studies of AI-based assessments in science (e.g., Maestrales et al, 2021;Wilson et al, 2023;Zhai, Shi, et al, 2021). Inequity could occur as a consequence of how teachers interpret and use the scores in their instructional decisions.…”

“…In science education, for decades researchers have employed scored students' written or drawn responses to train the machine to assign scores to new student responses (Ha et al, 2011;Haudek et al, 2011;Wang & Zhai, In press;Zhai, Yin, et al, 2020). Machine learning has been applied to automatically score students' understanding (Ha et al, 2011), explanations (Maestrales et al, 2021), and arguments (Wilson et al, 2023). Specifically, in our JRST publication , we used machine learning to automatically score students' drawn models and their written descriptions of models.…”

Success of university students in preparing hots questions using assignment-based direct instruction

“…Evaluation and student competence development work together to provide feedback on areas for potential realization and the effectiveness of the learning process [23]. Aspects of student competence include; (1) students' cognitive knowledge, i.e., thought processes and abilities; (2) students' knowledge; (3) students' ability (skill) to complete the assignments assigned to them [24]; (4) students' observation of desired values such as honesty, empathy, and openness; (5) students' attitudes, which shape their future trajectories; and (6) students' desire to undertake certain activities or behave in certain ways [25]. Ultimately, students must be able to apply these aspects of competence in their everyday lives.…”

Exploring Data Mining in Chemistry Education: Building a Web-Based Learning Platform for Learning Analytics

“…2 Within this development, learning analytics deals with analyzing learning process data to optimize the teaching processes, 3 for instance, by automating the analysis of learners' scientific argumentation. 4 Mining educational data opens the door to integrating artificial intelligence into teaching and learning processes and enables new approaches to address ongoing challenges in chemistry education: The COVID-19 pandemic has also demonstrated the need for remote learning tools for chemistry education. 5,6 A comprehensive analysis of the effective use of these educational resources can be enhanced by examining log files, especially when utilizing the provided digital features, which may not be immediately evident.…”

“…With the increasing technological progress in recent decades comes the exploration of artificial intelligence integrations in (science) education . Within this development, learning analytics deals with analyzing learning process data to optimize the teaching processes, for instance, by automating the analysis of learners’ scientific argumentation …”