Feasibility and psychometric properties of using peers, consulting physicians, co-workers, and patients to assess physicians

Violato, Claudio; Marini, Anthony; Toews, John; Lockyer, Jocelyn; Fidler, Herta

doi:10.1097/00001888-199710001-00028

Cited by 78 publications

(58 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The amount of rater experience did not change the results. Similar results have been reported by Kreiter and colleagues 16 (more than 8 ratings needed), Kwolek et al 31 (7-8 ratings), Ramsey et al 32 (11 ratings), Violato et al 33 (10 ratings), and Kroboth et al 34 (6-10 ratings). All reports agree that somewhere between 7 and 11 ratings are necessary to achieve a generalizable global estimate of competence when raters are basing ratings on a nonsystematic sample of observations.…”

Section: Accuracy In Estimating Competence (Estimating From a Discretsupporting

confidence: 87%

“…33,35,[54][55][56][57][58] These results suggest that either the individual competencies that make up clinical competence are highly correlated (i.e., individuals who perform one competency well are likely to perform well in the other competency areas); or that individual competencies are not highly correlated, but raters perceive them as such and rate them accordingly. Nathan and Lord 49 summarized their research on performance assessment by stating:…”

Section: Data Recallmentioning

confidence: 99%

See 1 more Smart Citation

SPECIAL ARTICLE: Cognitive, Social and Environmental Sources of Bias in Clinical Performance Ratings

Williams¹,

Klamen²,

McGaghie³

2003

Teaching and Learning in Medicine

349

250

View full text Add to dashboard Cite

show abstract

Section: Accuracy In Estimating Competence (Estimating From a Discretsupporting

confidence: 87%

Section: Data Recallmentioning

confidence: 99%

SPECIAL ARTICLE: Cognitive, Social and Environmental Sources of Bias in Clinical Performance Ratings

Williams¹,

Klamen²,

McGaghie³

2003

Teaching and Learning in Medicine

349

250

View full text Add to dashboard Cite

show abstract

“…[12][13][14][15][16][17] Its advantage lies in the synthesis of multiple perspectives to achieve a more stable, global rating. [18][19][20] An additional property is the ability to incorporate learner self-assessment in what is known as gap analysis. 16,18 Gap analysis examines the difference between the combined scores of ''expert'' faculty raters and the learner's self-score to obtain a measure of that individual's self-appraisal.…”

Section: Discussionmentioning

confidence: 99%

A Multirater Instrument for the Assessment of Simulated Pediatric Crises

Calhoun¹,

Boone²,

Miller³

et al. 2011

Journal of Graduate Medical Education

View full text Add to dashboard Cite

Background Few validated instruments exist to measure pediatric code team skills. The goal of this study was to develop an instrument for the assessment of resuscitation competency and self-appraisal using multirater and gap analysis methodologies. Methods Multirater assessment with gap analysis is a robust methodology that enables the measurement of self-appraisal as well as competency, offering faculty the ability to provide enhanced feedback. The Team Performance during Simulated Crises Instrument (TPDSCI) was grounded in the Accreditation Council for Graduate Medical Education competencies. The instrument contains 5 competencies, each assessed by a series of descriptive rubrics. It was piloted during a series of simulation-based interdisciplinary pediatric crisis resource management education sessions. Course faculty assessed participants, who also did self-assessments. Internal consistency and interrater reliability were analyzed using Cronbach α and intraclass correlation (ICC) statistics. Gap analysis results were examined descriptively. Results Cronbach α for the instrument was between 0.72 and 0.69. The overall ICC was 0.82. ICC values for the medical knowledge, clinical skills, communication skills, and systems-based practice were between 0.87 and 0.72. The ICC for the professionalism domain was 0.22. Further examination of the professionalism competency revealed a positive skew, 43 simulated sessions (98%) had significant gaps for at least one of the competencies, 38 sessions (86%) had gaps indicating self-overappraisal, and 15 sessions (34%) had gaps indicating self-underappraisal. Conclusions The TPDSCI possesses good measures of internal consistency and interrater reliability with respect to medical knowledge, clinical skills, communication skills, systems-based practice, and overall competence in the context of simulated interdisciplinary pediatric medical crises. Professionalism remains difficult to assess. These results provide an encouraging first step toward instrument validation. Gap analysis reveals disparities between faculty and self-assessments that indicate inadequate participant self-reflection. Identifying self-overappraisal can facilitate focused interventions.

show abstract

“…Thus, uptake of co-worker data for professional development purposes is predicated on reliable instrumentation. 15 The PAR instruments were psychometrically assessed with an examination of evidence for validity [16][17][18][19] and reliability [20][21][22][23][24][25] when they were first developed. While aspects of reliability have been reexamined for physician groups that have participated in PAR on more than one occasion, 12 there has not been a comprehensive examination of reliability for the co-worker instruments across all PAR specialties or specifically of reliability, a key component of validity.…”

Section: Co-worker Familiarity and Physician Multisource Feedbackmentioning

confidence: 99%

Co-Worker Assessment and Physician Multisource Feedback

2016

ARC Journal of Nursing and Healthcare

View full text Add to dashboard Cite

Abstract:Background: Multisource feedback (MSF) is increasingly being used as one of the components in revalidation and recertification processes to guide physicians' continuing professional development. Data provided by coworkers (e.g., nurses, pharmacists, technicians) are recognized as integral for assessing a physician's communication, teamwork and interprofessional abilities. The purpose of this study was to examine both the reliability of co-worker scores and the association between co-worker familiarity and physician ratings as both affect perceptions of the quality of feedback and the likelihood that recipients will take their feedback seriously.

show abstract

Feasibility and psychometric properties of using peers, consulting physicians, co-workers, and patients to assess physicians

Cited by 78 publications

References 0 publications

SPECIAL ARTICLE: Cognitive, Social and Environmental Sources of Bias in Clinical Performance Ratings

SPECIAL ARTICLE: Cognitive, Social and Environmental Sources of Bias in Clinical Performance Ratings

A Multirater Instrument for the Assessment of Simulated Pediatric Crises

Co-Worker Assessment and Physician Multisource Feedback

Contact Info

Product

Resources

About