Towards an Operational Definition of Clinical Competency in Pharmacy

Abstract

Objective. To estimate the inter-rater reliability and accuracy of ratings of competence in student pharmacist/patient clinical interactions as depicted in videotaped simulations and to compare expert panelist and typical preceptor ratings of those interactions. Methods. This study used a multifactorial experimental design to estimate inter-rater reliability and accuracy of preceptors' assessment of student performance in clinical simulations. The study protocol used nine 5-10 minute video vignettes portraying different levels of competency in student performance in simulated clinical interactions. Intra-Class Correlation (ICC) was used to calculate inter-rater reliability and Fisher exact test was used to compare differences in distribution of scores between expert and nonexpert assessments. Results. Preceptors (n542) across 5 states assessed the simulated performances. Intra-Class Correlation estimates were higher for 3 nonrandomized video simulations compared to the 6 randomized simulations. Preceptors more readily identified high and low student performances compared to satisfactory performances. In nearly two-thirds of the rating opportunities, a higher proportion of expert panelists than preceptors rated the student performance correctly (18 of 27 scenarios). Conclusion. Valid and reliable assessments are critically important because they affect student grades and formative student feedback. Study results indicate the need for pharmacy preceptor training in performance assessment. The process demonstrated in this study can be used to establish minimum preceptor benchmarks for future national training programs.

An Objective Structured Clinical Examination to Assess Competency Acquired During an Introductory Pharmacy Practice Experience

“…The OSCE provides a mechanism to standardize summative assessment across diverse practice sites and compare student performance across a single cohort. 1 In addition, these results can inform experiential office efforts concerning course design and preceptor development. Although alternatives, such as preceptor observation checklists, have been explored in the literature, these have produced a similar pattern of grade inflation to that seen with preceptor evaluations.…”

“…Student perceptions of the OSCEs were assessed using a standardized and validated institutional student course evaluation, specifically students' responses to the statement: "The exams were representative of materials and objectives presented in the course." Students were allowed to rank this statement on a five-point scale with anchors of strongly agree (5) and strongly disagree (1). The survey, which was voluntary and de-identified, was administered to students upon completion of the course but before grades were released.…”

“…In particular, preceptor ratings of student performance have been criticized for their lack of consistency and accuracy. [1][2][3] Nevertheless, a majority of pharmacy schools continue to use preceptor evaluations as the primary source for grade determination in both IPPEs and APPEs. 4 One alternative to the traditional preceptor evaluation is the objective structured clinical examination (OSCE).…”

“…Considering that key drivers of preceptor rater error include leniency, deficits in preceptor knowledge and skills, and the halo effect of a students' prior performance, OSCE lends itself as a more objective and unbiased alternative. 1 Use of the OSCE in pharmacy curricula is common, but varies greatly in structure and purpose. 6 Schools of pharmacy have implemented OSCEs as formative and summative assessments for both individual courses and program level performance, and most recently to assess students' readiness to begin APPEs.…”