Background A common feature of performance assessments is the use of human assessors to render judgements on student performance. From a measurement perspective, variability among assessors when assessing students may be viewed as a concern because it negatively impacts score reliability and validity. However, from a contextual perspective, variability among assessors is considered both meaningful and expected. A qualitative examination of assessor cognition when assessing student performance can assist in exploring what components are amenable to improvement through enhanced rater training, and the extent of variability when viewing assessors as contributing their individual expertise. Therefore, the purpose of this study was to explore assessor cognition as a source of score variability in a performance assessment of practice-based competencies. Method A mixed-method sequential explanatory study design was used where findings from the qualitative strand assisted in the interpretation of results from the quantitative strand. Scores from one objective structured clinical examination (OSCE) were obtained for 95 occupational therapy students. Two Generalizability studies were conducted to examine the relative contribution of assessors as a source of score variability and to estimate the reliability of domain and holistic scores. Think-aloud interviews were conducted with eight participants assessing a subset of student performances from the OSCE in which they participated. Findings from the analysis of think-aloud data and consideration of assessors’ background characteristics were used to assist in the interpretation of variance component estimates involving assessors, and score reliability. Results Results from two generalizability analyses indicated the highest-order interaction-error term involving assessors accounted for the second-highest proportion of variance, after student variation. Score reliability was higher in the holistic vs. analytic scoring framework. Verbal analysis of assessors' think-aloud interviews provided evidential support for the quantitative results. Conclusions This study provides insight into the nature and extent of assessor variability during a performance assessment of practice-based competencies. Study findings are interpretable from the measurement and contextual perspectives on assessor cognition. An integrated understanding is important to elucidate the meaning underlying the numerical score because the defensibility of inferences made about students’ proficiencies rely on score quality, which in turn relies on expert judgements.

No abstract

Background Using tools from outside healthcare can help improve patient safety. Pointing and Calling (Shisa Kanko) is an operational procedure developed for industry in Japan to prevent human error and has been used in healthcare in Asian countries to reduce errors during medication administration. Pointing and Calling affects cognitive task switching by pointing to a place or object and calling out the operation to be performed. Aim Conduct an initial use case to examine the willingness and ability of healthcare professionals in a Western country to use Pointing and Calling. Methods An observational initial use case was conducted with nineteen Advanced Care Paramedic students. Confidence, perceptions, and use of Pointing and Calling were measured during a simulated clinical scenario along with facilitator perceptions. Results After the simulation participants were confident in their ability to use Pointing and Calling, found the method to be beneficial, and indicated they would use Pointing and Calling in the future. Participants often used the method for tasks such as checking vitals. Aspects of the method requiring clarification and more training were identified. Facilitators indicated the method appeared beneficial during simulations and could be incorporated into existing curriculum. Conclusions The benefits of Pointing and Calling are readily apparent to students and facilitators and both groups are receptive to the method. Pointing and Calling is low risk with substantial potential benefits. With more education and training Pointing and Calling could be effectively implemented.

This critical review aimed to synthesize the literature and critique the strength of the methodological quality of current evidence regarding examiner bias related to ethnicity and gender in objective structured clinical examination implemented in health professions education. The Guidelines for Critical Review (GCR) was used to critically appraise the selected studies. Ten studies were retrieved for review. The overall quality of the papers was moderate. Two studies met all the criteria of the GCR, indicating stronger evidence of their outcomes. One of them reported ethnic and gender bias potentially existing, while another found only one examiner showing consistent ethnic bias. No systematic bias was found across the studies. Nonetheless, the possibility of ethnic or gender bias by some examiners cannot be ignored. To mitigate potential examiner bias, the investigation of implicit bias training, frame of reference training, the use of multiple examiners, and combination assessments are suggested.

No abstract