Reliability of the Interprofessional Collaborator Assessment Rubric (ICAR) in Multi Source Feedback (MSF) with post-graduate medical residents

Abstract

BackgroundIncreased attention on collaboration and teamwork competency development in medical education has raised the need for valid and reliable approaches to the assessment of collaboration competencies in post-graduate medical education. The purpose of this study was to evaluate the reliability of a modified Interprofessional Collaborator Assessment Rubric (ICAR) in a multi-source feedback (MSF) process for assessing post-graduate medical residents’ collaborator competencies.MethodsPost-graduate medical residents (n = 16) received ICAR assessments from three different rater groups (physicians, nurses and allied health professionals) over a four-week rotation. Internal consistency, inter-rater reliability, inter-group differences and relationship between rater characteristics and ICAR scores were analyzed using Cronbach’s alpha, one-way and two-way repeated measures ANOVA, and logistic regression.ResultsMissing data decreased from 13.1% using daily assessments to 8.8% utilizing an MSF process, p = .032. High internal consistency measures were demonstrated for overall ICAR scores (α = .981) and individual assessment domains within the ICAR (α = .881 to .963). There were no significant differences between scores of physician, nurse, and allied health raters on collaborator competencies (F2,5 = 1.225, p = .297, η2 = .016). Rater gender was the only significant factor influencing scores with female raters scoring residents significantly lower than male raters (6.12 v. 6.82; F1,5 = 7.184, p = .008, η 2 = .045).ConclusionThe study findings suggest that the use of the modified ICAR in a MSF assessment process could be a feasible and reliable assessment approach to providing formative feedback to post-graduate medical residents on collaborator competencies.

Charting a Key Competency Domain: Understanding Resident Physician Interprofessional Collaboration (IPC) Skills

“…14 We therefore created a behaviorally defined assessment tool, based on Interprofessional Education Collaborative domains and competencies, and tailored to the routine physician-nurse interactions that occur frequently as part of primary care practice for common chronic (e.g., diabetes, hypertension) and urgent conditions. Additional tools focused on assessing performance have since been developed (see, for example, Interprofessional Collaborator Assessment Rubric 15 and the Performance Assessment for Communication and Teamwork 16 ), and while these share many of the core skills and domains we identified from the literature, they also focus on the team and not individuals, including domains such as situational monitoring, team goals and use of protocols/ checklists not relevant to PC scenarios, and tend to use judgment-based scoring options (e.g., below expectations or excellent). Our OSCE assessments are designed to help Standardized Patients rate consistently and accurately by using case-specific behavioral descriptors (observable actions) as the basis for selecting scoring options.…”

Development of a Standardized Kalamazoo Communication Skills Assessment Tool for Radiologists: Validation, Multisource Reliability, and Lessons Learned

“…Ratings may also have been influenced by how well reviewers knew those being assessed, sex differences, and differences in radiologists' and nonradiologists' personal or professional expectations of radiologists. Other studies conducted with multisource feedback in medical education, including within radiology, have shown significant differences in competency evaluations between assessor groups [7,21,22]. Nonetheless, our use of disparate representative stakeholders as reviewers and raters was consonant with beliefs that interdisciplinary collaboration is optimal for assessing competency in communication and interpersonal skills and that those with expertise in humanistic and psychosocial aspects of health care bring uniquely valuable insights into the evaluation process [11,15,22,23].…”

“…Other studies conducted with multisource feedback in medical education, including within radiology, have shown significant differences in competency evaluations between assessor groups [7,21,22]. Nonetheless, our use of disparate representative stakeholders as reviewers and raters was consonant with beliefs that interdisciplinary collaboration is optimal for assessing competency in communication and interpersonal skills and that those with expertise in humanistic and psychosocial aspects of health care bring uniquely valuable insights into the evaluation process [11,15,22,23]. Such an interdisciplinary approach may be particularly important for assessing communication and relational skills in radiology, in which no established standards for excellence exist, few validated communication skills programs have been developed, and faculty development remains largely ad hoc.…”

“…Such an interdisciplinary approach may be particularly important for assessing communication and relational skills in radiology, in which no established standards for excellence exist, few validated communication skills programs have been developed, and faculty development remains largely ad hoc. The use of multisource feedback for assessing clinical competencies has been further endorsed for its recognition that health care delivery is becoming increasingly interdisciplinary and team based and that this trend requires assessors from a variety of perspectives [11,15,22,24]. This is of substantial relevance to radiology, the value of which is becoming increasingly tied to its embeddedness within disparate health care system domains [25][26][27][28].…”

Leadership, Management and Team Competencies of Filipino Nursing Student Manager-Leaders: Implications on Nursing Education

“…Multi-source evaluation -A supervisor-, subordinate-and self-appraisal -was used in this study. Available literature found potential value in the use of a 360-degree or multi-source evaluation particularly among nurses [26] and post-graduate medical residents [27]. Variance in the ratings from a multisource appraisal cannot be attributed to rater groups since the same underlying constructs are being measured in the study [28].…”