“Who writes what?” Using written comments in team-based assessment to better understand medical student performance: a mixed-methods study

Abstract

BackgroundObservation of the performance of medical students in the clinical environment is a key part of assessment and learning. To date, few authors have examined written comments provided to students and considered what aspects of observed performance they represent. The aim of this study was to examine the quantity and quality of written comments provided to medical students by different assessors using a team-based model of assessment, and to determine the aspects of medical student performance on which different assessors provide comments.MethodsMedical students on a 7-week General Surgery & Anesthesiology clerkship received written comments on ‘Areas of Excellence’ and ‘Areas for Improvement’ from physicians, residents, nurses, patients, peers and administrators. Mixed-methods were used to analyze the quality and quantity of comments provided and to generate a conceptual framework of observed student performance.Results1,068 assessors and 127 peers provided 2,988 written comments for 127 students, a median of 188 words per student divided into 26 “Areas of Excellence” and 5 “Areas for Improvement”. Physicians provided the most comments (918), followed by patients (692) and peers (586); administrators provided the fewest (91). The conceptual framework generated contained four major domains: ‘Student as Physician-in-Training’, ‘Student as Learner’, ‘Student as Team Member’, and ‘Student as Person.’ConclusionsA wide range of observed medical student performance is recorded in written comments provided by members of the surgical healthcare team. Different groups of assessors provide comments on different aspects of student performance, suggesting that comments provided from a single viewpoint may potentially under-represent or overlook some areas of student performance. We hope that the framework presented here can serve as a basis to better understand what medical students do every day, and how they are perceived by those with whom they work.

Analysis of dental students' written peer feedback from a prospective peer assessment protocol

“…Amongst similar reported themes from our subjects, participants in the Nofziger et al study remembered having received feedback on ‘being too quiet’, ‘harder challenges’, ‘overconfidence’, ‘knowing your limits’, ‘low self‐esteem’ and ‘low self‐confidence’. Another study of third‐year medical students, this time by White and Sharma , reported similar written peer feedback themes as our clinical BDS 5 students. Amongst these, they mentioned information management, initiative and self‐direction, asking questions to learn, confidence, compassion and respect for others.…”

Does what we write matter? Determining the features of high- and low-quality summative written comments of students on the internal medicine clerkship using pile-sort and consensus analysis: a mixed-methods study

“…This is slightly lower than other published studies in which 70–94 % of analyzed comments were coded as positive [ 6 , 7 , 9 – 11 , 32 , 33 ]. The lower rate of positive comments is in line with other studies of faculty written comments to third-year clerkship students [ 9 ]; faculty written comments to residents [ 6 , 10 , 11 ] and to peers tend to be more markedly positive [ 33 ]. The perception that a comment was helpful was not associated with the valence of the comment – Cluster A, the highly helpful comments, had the same percent of positive comments as Cluster D, the least highly helpful comments.…”

“…Research into the meaning of written comments about learners has used a single type of investigation - faculty analysis of the content of comments written by other faculty. These studies use written comments about third-year medical students [ 1 , 3 , 5 , 7 – 9 ] or residents [ 6 , 10 – 12 ] and code comments according to polarity (positive/negative) and internally-created subject categories, with category subjects varying from study to study and including such topics as “knowledge,” [ 10 ] “personal characteristics,” [ 1 ] “work ethic,” [ 9 ] “behavior,” [ 6 ] “future life as a physician,” [ 3 ] and “initiative” [ 7 ]. From this body of literature we learn that written comments are infrequently related to clinical skills [ 1 ] but often related to professional behaviors [ 7 ].…”

“…From this body of literature we learn that written comments are infrequently related to clinical skills [ 1 ] but often related to professional behaviors [ 7 ]. Overall, the majority of written comments have a positive polarity [ 3 , 7 , 9 – 11 ], but comments with negative polarity provide more discriminating information [ 3 ] and are viewed more seriously than those with positive or neutral polarity [ 12 ]. Faculty deem the quality of written comments given to students variable [ 1 , 8 , 11 ].…”

Seeing but not believing: Insights into the intractability of failure to fail

“…49 In the context of clinical supervision, the unexpectedness of encountering insufficient engagement with learning may signal a contradictory supervisory action and evoke dissonance. write about engagement with the learning process in their comments, 54,55 although such comments can be interpreted negatively by others. 56 Therefore, strategies to counter associated stigma may be needed to implement assessments specifically designed to document learning process engagement.…”