Automated Analysis of Middle School Students’ Written Reflections During Game-Based Learning

Game-Based Learning Analytics for Supporting Adolescents’ Reflection

“…These prompts were triggered when participants completed actions deemed critical for achieving the learning goals (see Table 1 for details). Results showed that the average reflection quality across the three reflection types was positively related to learning outcomes (Carpenter et al, 2020). This study highlights the fact that examining the quality with which adolescents reflect provides insight into the role of reflection on learning with GBLEs, but the authors failed to consider the other learning goal, i.e., solving the mystery to study how reflection quality impacted successful problem solving.…”

“…They also found that learners in the open-prompt condition did not significantly differ from those in the no-prompt condition (Johnson & Mayer, 2010). It is surprising that no effect of open-ended reflection on learning was found since previous studies have found that open-ended written reflection responses reveal reflection depth that relates to higher learning outcomes (Wu & Looi, 2012;Carpenter et al, 2020;Ullmann, 2019). Additional challenges exist, since the simple act of selecting reasons for initiating an action does not necessarily elicit reflection, drawing us to question whether reflection played a role in the differences between the conditions.…”

“…For instance, can we ensure that the learners were reflecting solely based the prompt they received and the number of times they completed a written response, i.e., the quantity of written reflection? Carpenter et al (2020) demonstrated a novel technique to gauge the quality of reflection by examining adolescents' written reflection responses using machine learning to automatically detect the quality at which learners reflected and assess its relation to learning with Crystal Island, a GBLE designed with two learning goals: learn about microbiology concepts and solve a mysterious illness on the island . Reflection quality was defined based on the depth of the written response, with emphasis placed on whether the participant was observing, contemplating, and inspecting their thoughts, ideas, beliefs, and perspectives as they related to achieving a learning goal and whether they generated a clear hypothesis (i.e., beliefs about a particular direction for their problem solution or learning) to inform their next set of actions.…”

“…For this very reason, we argue that a model of reflection that accounts for the goals or objectives built into the learning environment needs to be adopted in order to gain insight into whether learners were observing, contemplating, and inspecting their beliefs, actions, thoughts, etc., in relation to achieving their learning goals. As highlighted in the discussion above, gaps remain in the literature about whether different reflection-prompt types (e.g., open, directed) impact adolescents' ability to practise reflection and how the quality of their reflections relates to learning and problem solving as they align with particular learning goals and objectives (Carpenter et al, 2020;Geden et al, 2020;Vrugte et al, 2015;Johnson & Mayer, 2010;Fiorella & Mayer, 2012). To address these challenges, we guided our research using McAlpine et al (1999)'s model of reflection relative to other models since it emphasizes that learning goals guide reflection, which impacts reasoning, decision making, monitoring, and knowledge acquisition.…”

Investigating Student Reflection during Game-Based Learning in Middle Grades Science

“…To assess the depth of students' reflections during their interaction with Crystal Island, a rubric that one of the authors previously helped to develop [4] was used (see Table 2). This rubric scores reflection depth on a scale from 1 (not reflective) to 5 (highly reflective).…”