Assessing the Reliability of Performance Assessment Scores: Some Considerations in Selecting an Appropriate Framework

Champlain, André F. De; Gotzmann, Andrea; Qin, Sirius

doi:10.4300/jgme-d-15-00751.1

Cited by 2 publications

(1 citation statement)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Most studies aimed at assessing the reliability of MSF ratings have reported Cronbach alpha values as an index which is unsuitable as it cannot properly partition and account for all sources of measurement error (Lee et al 2016;Lockyer et al 2007;Overeem et al 2012;Violato et al 2003;Zhao et al 2013). This can lead to instances where the reliability coefficient is spuriously inflated (De Champlain et al 2016). Other studies correctly employed generalizability theory as a broader framework to not only assess the reliability of MSF ratings but also to quantify the impact of various sources of measurement error (Wright et al 2012;Archer et al 2005;Campbell et al 2008Campbell et al ,2010Crossley 2015;Garra et al 2011;Moonen-van Loon et al 2015;Violato et al 2008).…”

Section: Psychometric Evidencementioning

confidence: 99%

A review of multi-source feedback focusing on psychometrics, pitfalls and some possible solutions

2021

Self Cite

View full text Add to dashboard Cite

With the re-emergence of competency-based frameworks in professional education, multisource feedback (MSF) has become a common method for assessing various competencies, including communication, professionalism, and aspects of teambased performance. A wide variety of publications over the past 50 years or more in the business and health literature would seem to support the use of MSF at least for quality improvement (QI) purposes. However, our own experience with using MSF in physicians has been quite mixed, with some physicians embracing the experience and making improvements and others the complete opposite. We decided to review the existing literature on MSF to try to identify key aspects of successful MSF programs. This paper presents a structured critique of the literature on the use of MSF in physician populations. The findings were surprising as key assumptions around the validity and reliability of MSF were not consistently met, key lessons from earlier research were not carried over to present day programs and perhaps most concerning was a lack of evidence for MSF producing meaningful sustained behavior change. From these findings we suggest some key areas of potential improvement in MSF programs.

show abstract