Machine Learning Mixed Methods Text Analysis: An Illustration From Automated Scoring Models of Student Writing in Biology Education

Abstract

Assessing student knowledge based on their writing using traditional qualitative methods is time-consuming. To improve speed and consistency of text analysis, we present our mixed methods development of a machine learning predictive model to analyze student writing. Our approach involves two stages: first an exploratory sequential design, and second an iterative complex design. We first trained our predictive model using qualitative coding of categories (ideas) in student writing. We next revised our model based on feedback from instructor-users. The model itself highlighted categories in need of revision. The contribution to mixed methods research lies in our innovative use of the machine learning tool as a rapid, consistent additional coder, and a resource that can predict codes for new student writing.

Extending a Pretrained Language Model (BERT) using an Ontological Perspective to Classify Students’ Scientific Expertise Level from Written Responses

“…Rubric development began by reviewing examples of published rubrics that were used in similar assessments and intended for use with automatic scoring (Jescovitch et al, 2021;Sripathi et al, 2023). We agreed upon a scale that would best represent the students' varying levels of knowledge (Table 2).…”

No, Let's Not Go Back to Handwritten Activities: Inclusive Teaching Strategies in the Context of ChatGPT

“…Machine learning technologies, LLMs, and ChatGPT have opportunities for application on both “sides” of teaching, serving as potential resources related to student use and to interactive instruction or feedback. Particularly related to feedback, the potential for formative assessment provides a significant opportunity for instructors to enhance their understanding of student ideas and thinking (Sripathi et al, 2017, 2023). This can include automated assessment of student constructed (written) responses in formative feedback, as compared to multiple choice “clicker” questions (Musib et al, 2017).…”

No, Let's Not Go Back to Handwritten Activities: Inclusive Teaching Strategies in the Context of ChatGPT

“…Machine learning technologies, LLMs, and ChatGPT have opportunities for application on both “sides” of teaching, serving as potential resources related to student use and to interactive instruction or feedback. Particularly related to feedback, the potential for formative assessment provides a significant opportunity for instructors to enhance their understanding of student ideas and thinking (Sripathi et al, 2017; 2023). This can include automated assessment of student constructed (written) responses in formative feedback, as compared to multiple choice “clicker” questions (Musib et al, 2017).…”