Grading reflective essays: the reliability of a newly developed tool- GRE-9

Abstract

Background The main objective of this study is the development of a short reliable easy-to-use assessment tool in the aim of providing feedback to the reflective writings of medical students and residents. Methods This study took place in a major tertiary academic medical center in Beirut, Lebanon. Seventy-seven reflective essays written by 18 residents in the department of Family Medicine at the American University of Beirut Medical Center (AUBMC) were graded by 3 raters using the newly developed scale to assess the scale reliability. Following a comprehensive search and analysis of the literature, and based on their experience in reflective grading, the authors developed a concise 9-item scale to grade reflective essays through repeated cycles of development and analysis as well as the determination of the inter-rater reliability (IRR) using intra-class correlation coefficients (ICC) and Krippendorff’s Alpha. Results The inter-rater reliability of the new scale ranges from moderate to substantial with ICC of 0.78, 95% CI 0.64–0.86, p < 0.01 and Krippendorff’s Alpha was 0.49. Conclusions The newly developed scale, GRE-9, is a short, concise, easy-to-use reliable grading tool for reflective essays that has demonstrated moderate to substantial inter-rater reliability. This will enable raters to objectively grade reflective essays and provide informed feedback to residents and students.

Grading Reflective Essays: The Construct Validity and Reliability of a Newly Developed Tool- GRE-9

“…As yielded in the primary study, content-related evidence was supported by the theory-informed construction of the study instrument since the GRE-9 rubric was based on a comprehensive analysis of relevant theoretical models of re ection as well as existing re ection assessment measures (Wald, Davis, Reis, Monroe & Borkan, 2009). In addition to content validity, GRE-9 was found to be a reliable, concise and simple grading tool that has demonstrated moderate to substantial inter-rater reliability (Makarem et al, 2020), yet the investigation of the construct validity was not determined due to the small sample size. As such, given that clear robust validity of an instrument is crucial (Lee & Clarke, 2015), the present study aimed to further investigate the psychometrics of the GRE-9 by examining its construct validity, intra rater reliability and internal consistency.…”

“…The developed GRE-9 was found to be a reliable, concise and simple grading tool that has demonstrated moderate to substantial inter-rater reliability enabling raters to objectively grade re ective essays and provide informed feedback to residents and students (Makarem et al, 2020).…”

A systematic scoping review of reflective writing in medical education

“…This extends beyond Thorpe [ 96 ]’s study which categorises reflective efforts into ‘non-reflectors’, ‘reflectors’, ‘critical reflectors’, and focuses on their process, structure, depth and content. van Manen [ 97 ], Plack et al [ 98 ], Rogers et al [ 99 ] and Makarem et al [ 100 ] begin with evaluating the details of the events. Kim’s Critical Reflective Inquiry Model [ 85 ] and Bain’s 5Rs Reflective Framework [ 101 ] also consider characterisations of emotions involved.…”

“…Other models appraise the intentions behind actions and thoughts [ 85 ], the factors precipitating the event [ 101 ] and meaning-making [ 85 ]. Other theories consider links with previous experiences [ 100 ], the integration of thoughts, justifications and perspectives [ 99 ], and the hypothesising of future strategies [ 98 ].…”

“…Framing of reflection into local, regional, national and global context Morrow's Critical Reflection [ 107 ] Personal Interpersonal Contextual Critical/ Evaluation – limitations faced, social, ethical problems faced Plack et al's Method of Assessing Reflective Journal Writing [ 108 ] No evidence of reflection Evidence of reflection Evidence of critical reflection i.e. exploration of existence of problem, where problems arises from, underlying assumptions; revisits experience to challenge assumptions and modification of biases Kims’s Critical Reflective Inquiry Model [ 85 ] Descriptive Description of practice events, actions, thoughts and feelings Reflective Analysis of situation, of intentions Critical Critique of practice regarding conflicts, distortion and inconsistencies Engagement in emancipatory change process Makaram et al's GRE-9 [ 100 ] What happened? What is special about this event?…”