Towards Reflective Writing Analytics: Rationale, Methodology and Preliminary Results

Abstract

When used effectively, reflective writing tasks can deepen learners' understanding of key concepts, help them critically appraise their developing professional identity, and build qualities for lifelong learning. As such, reflective writing is attracting substantial interest from universities concerned with experiential learning, reflective practice, and developing a holistic conception of the learner. However, reflective writing is for many students a novel genre to compose in, and tutors may be inexperienced in its assessment. While these conditions set a challenging context for automated solutions, natural language processing may also help address the challenge of providing real time, formative feedback on draft writing. This paper reports progress in designing a writing analytics application, detailing the methodology by which informally expressed rubrics are modelled as formal rhetorical patterns, a capability delivered by a novel web application. Preliminary tests on an independently human-annotated corpus are encouraging, showing improvements from the first to second version, but with much scope for improvement. We discuss a range of issues: the prevalence of false positives in the tests, areas for future technical improvements, the issue of gaming the system, and the participatory design process that has enabled work across disciplinary boundaries to develop the prototype to its current state.

Evaluating Machine Learning Approaches to Classify Pharmacy Students’ Reflective Statements

“…Chen et al [20] adapted topic modelling to analyze pre-service teachers' reflection journals, but topic models focus on the content rather than quality and depth of reflection. Extending the work of Buckingham Shum et al [4], Gibson et al [3] proposed a concept-matching rhetorical analysis framework [21] to automatically detect sentences performing three key reflective rhetorical functions summarized as Context, Challenge and Change. Kovanovic et al [15] developed a random forest classifier using features extracted from the LIWC and Coh-Metrix for arts students' reflective statements (observation, motive and goal).…”

“…Moreover, finding ways to scaffold quality reflection becomes all the more important as we seek to provide learners with more authentic tasks and assessments, that are distinctive because of the rich, complex, social, psychological and embodied experiences they provide [2]. When students engage effectively in reflective learning activities, this can provide evidence of self-critical insight, identify challenging issues, connect academic with experiential knowledge, acknowledge emotions and feelings, and reflect on how they can apply such insights in the future [3,4]. Reflective processes in learning have most impact when they are formative and future-oriented [5].…”

Analytics-enabled teaching as design

“…In these cases, learning analytics require sensemaking on part of the teacher before some kind of actionable change occurs in the educational setting. In other cases, learning analytics can also be hardwired into the learning or support system -for example, in reflective writing analytics [e.g., [42][43], or adaptive computer-supported learning [e.g., 44], or personalised, recommender systems for learning [e.g., 45]. Some of these designs, for instance, reflective writing practice with analytics, can also offer compounding benefits such as developing learner metacognition for ethical and cultural competence [46].…”

Utilizing an Automated Tool Analysis to Evaluate EFL Students’ Writing Performances

“…The uses and effects of automated tools in analyzing students' writing abilities have been investigated for several years (Haswell, 2000;Buckingham Shum et al, 2017, Ullmann, 2019. However, recent advances in computational linguistics and discourse processing have shown possibilities that educators or researchers can automate many language-and text-processing mechanisms.…”