Assessing Communication Skills of Medical Students in Objective Structured Clinical Examinations (OSCE) - A Systematic Review of Rating Scales

Abstract

BackgroundTeaching and assessment of communication skills have become essential in medical education. The Objective Structured Clinical Examination (OSCE) has been found as an appropriate means to assess communication skills within medical education. Studies have demonstrated the importance of a valid assessment of medical students’ communication skills. Yet, the validity of the performance scores depends fundamentally on the quality of the rating scales used in an OSCE. Thus, this systematic review aimed at providing an overview of existing rating scales, describing their underlying definition of communication skills, determining the methodological quality of psychometric studies and the quality of psychometric properties of the identified rating scales.MethodsWe conducted a systematic review to identify psychometrically tested rating scales, which have been applied in OSCE settings to assess communication skills of medical students. Our search strategy comprised three databases (EMBASE, PsycINFO, and PubMed), reference tracking and consultation of experts. We included studies that reported psychometric properties of communication skills assessment rating scales used in OSCEs by examiners only. The methodological quality of included studies was assessed using the COnsensus based Standards for the selection of health status Measurement INstruments (COSMIN) checklist. The quality of psychometric properties was evaluated using the quality criteria of Terwee and colleagues.ResultsData of twelve studies reporting on eight rating scales on communication skills assessment in OSCEs were included. Five of eight rating scales were explicitly developed based on a specific definition of communication skills. The methodological quality of studies was mainly poor. The psychometric quality of the eight rating scales was mainly intermediate.DiscussionOur results reveal that future psychometric evaluation studies focusing on improving the methodological quality are needed in order to yield psychometrically sound results of the OSCEs assessing communication skills. This is especially important given that most OSCE rating scales are used for summative assessment, and thus have an impact on medical students’ academic success.

Screening for oropharyngeal dysphagia in older adults: A systematic review of self‐reported questionnaires

“…Evaluating the quality of the psychometric properties of the instruments identified was performed by applying the Terwee et al quality criteria following the model of analysis performed in another study, as presented in Table .…”

“…After selecting the articles, an analysis was performed by the COSMIN (COnsensus based Standards for the selection of health Measurement INstruments), a standardised guideline for critical evaluation of the methodological quality of studies that investigate the psychometric properties of measurement instruments in health. COSMIN became known in 2006, when a group of researchers published this initiative to standardise analysis of the psychometric properties of self‐reported questionnaires, and its product resulted in a checklist that describes a set of criteria for evaluating measures in health care, as well as a description of how results may be interpreted on a scale of four categories based on the interpretation strategies of recent systematic reviews …”

“…The psychometric evaluation quality of instruments was subsequently re‐examined in the following sequence: content validity, internal consistency, criterion validity, construct validity and reproducibility (compliance and reliability), responsiveness, floor and ceiling effects and interpretability, by evaluating the presence or not of these properties using the following categories of evaluation: positive (+), undetermined (? ), negative (−) or no information available (0) …”

“…COSMIN became known in 2006, when a group of researchers 26 published this initiative to standardise analysis of the psychometric properties of self-reported questionnaires, and its product resulted in a checklist that describes a set of criteria for evaluating measures in health care, 22 as well as a description of how results may be interpreted on a scale of four categories 23 based on the interpretation strategies of recent systematic reviews. [27][28][29][30] COSMIN was first used to assess whether the screening instruments of studies fit into the patterns of good methodological quality; and secondly, 11 psychometric properties were arranged in a table, namely internal consistency (A), reliability (B), measurement error (C), content validity (D), structural validity (E), hypotheses testing (F), cross-cultural validity (G), criterion validity (H), responsiveness (I), interpretability (J) and generalisability (K). 22,31 Each of these properties is composed by a set of five to 18 items for the included article to be analysed, which were assessed according to the scale of four categories established by, 23 in which the evaluators should score as: excellent (+++), good (++), fair (+) or poor (0).…”

“…), negative (−) or no information available (0). [28][29][30]32 For best practice training and calibration to ensure consistent implementation of the COSMIN checklist, and according to the scale of categorisation described by Terwee et al, 32 two reviewers applied the protocol to an article 33 of self-reported oropharyngeal dysphagia questionnaire in older people who did not fit the criteria for eligibility and which was translated and adapted for other languages. 34,35 Thus, the evaluators discussed their results for establishing criteria for tie breaking between their assessments when there was any disagreement or ambiguity in interpreting the criteria applied to the article.…”