Automated Guidance for Thermodynamics Essays: Critiquing Versus Revisiting

Abstract

Middle school students struggle to explain thermodynamics concepts. In this study, to help students succeed, we use a natural language processing program to analyze their essays explaining the aspects of thermodynamics and provide guidance based on the automated score. The 346 sixth-grade students were assigned to either the critique condition where they criticized an explanation or the revisit condition where they reviewed visualizations. Within each condition, the student was assigned one of two types of tailored guidance based on the sophistication of their original essay. Both forms of guidance led to significant improvement in student understanding on the posttest. Guidance was more effective for students with low prior knowledge than for those with high prior knowledge (consistent with regression toward the mean). However, analysis of student responses to the guidance illustrates the value of aligning guidance with prior knowledge. All students were required to revise their essay as an embedded assessment. While effective, teachers involved in this study reported that revising is resisted by students and does not align with typical, vocabulary-focused classroom writing activities.

Enhancing student explanations of evolution: Comparing elaborating and competing theory prompts

“…Since both conditions benefitted from knowledge integration guidance, this is understandable. Consistent with prior research on knowledge integration (and regression towards the mean), students with low prior knowledge learned significantly more than those with high prior knowledge (Donnelly et al, ). These results show the benefit of guidance for low prior knowledge students.…”

“…The majority of the coding was conducted by the second and fourth author, and the first author facilitated discussion on resolving disagreements in the coding. Consistent with previous research involving the use of KI rubrics (Donnelly, Vitale, & Linn, ), interrater agreement was generally high for the second and fourth authors. The interrater agreements for the items were 82.7% for the “Giraffe” item (κ = 0.70, 784 responses), 81.1% for the “Fruitfly” item (κ = 0.67, 784 responses), 80.3% for the “Guppies” item (κ = .68, 784 responses), 91.3% for the “Global Climate Change” item (κ = 0.86, 764 responses), and 94.3% for the “Explain Evolution” item (κ = .81, 392 responses).…”

Fostering Fifth-Grade Students’ Ability to Design Comparative Experiments Using an Online Model

“…The Thermodynamics Challenge WISE unit (http://wise.berkeley.edu/project/20760#/vle/node20) designed by researchers at UC Berkeley was translated into Chinese and modified according to fifth-grade students' learning capability (Donnelly, Vitale, & Linn, 2015). During the unit, students were asked to choose suitable materials for designing hot and cold beverages containers.…”

Enhancing writing analytics in science education research with machine learning and natural language processing—Formative assessment of science and non-science preservice teachers’ written reflections

“…10.3389/feduc.2022 Frontiers in Education 04 frontiersin.org ML and NLP have been adopted in a variety of contexts in science education research with written language artifacts. Science education researchers used ML models to assess complex constructs in constructed responses (Ha et al, 2011;Wulff et al, 2020;Zhai et al, 2022), explore patterns in large language-based datasets (Odden et al, 2021;Wulff P. et al, 2022), or develop means for automated guidance and feedback (Donnelly et al, 2015;Zhai et al, 2020). In these studies, it has been shown that ML and NLP could reliably and validly classify argumentations, explanations, or reflections of middle-school, high-school, and university students and teachers.…”