Introduction Reflection and self‐assessment are critical skills for healthcare providers. Identification of gaps in knowledge, skills and attitudes, along with the ability to critically think and problem solve to fill gaps, is the ultimate outcome for lifelong learning. The aims of this study were to (a) refine an instrument used for measuring reflective ability, and conduct comprehensive reliability testing, and (b) describe a process for rater calibration. Materials and Methods Students develop e‐portfolios over a four‐year span with assignments that require reflection and self‐assessment. The final piece of the portfolio includes a global reflection written the last semester of the programme. Three faculty raters independently evaluated 106 dental students’ global reflections using the revised grading rubric. An intraclass correlation coefficient measured the level of agreement between the three raters. Results Analysis of the 318 faculty ratings (106/rater) resulted in an intraclass correlation of .708. Based on a 5‐point grading scale (0 = does not respond to the assignment to 5 = reconstructing), the ratings of the 106 global reflections ranged from 1.3 to 5.0 (M = 3.1, SD =0.66). Discussion This study provides confidence in the reliability of a grading rubric designed to assess reflective ability, along with suggestions for calibration. An overall mean of 3.1 (Level 3 = relating—includes evidence of lessons learned) illustrates the complexity of teaching reflection and self‐assessment. Conclusion Use of a reliable grading rubric for assessing reflective writing could assist schools interested in incorporating reflection and self‐assessment into the curriculum, ultimately supporting lifelong and enhanced health care.