Assessment of competence in clinical reasoning and decision‐making under uncertainty: the script concordance test method

Ramaekers, Stephan; Kremer, W.D.J.; Pilot, Albert; Beukelen, Peter van; Keulen, Hanno van

doi:10.1080/02602938.2010.500103

Cited by 20 publications

(28 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Such transient errors cause item errors to be correlated, inflating coefficient alpha . The few SCT studies reviewed that actually administered SCTs on multiple occasions make it apparent that SCT test–retest errors are far from trivial …”

Section: Reliability and Internal Structure Of Sctsmentioning

confidence: 54%

“…Of the seven studies reporting a statistic other than coefficient alpha, two studies used generalisability theory analyses focused only on case or item facets and three studies estimated test–retest reliability, with correlations varying from r = 0.02 to r = 0.76 . Only one study conducted a generalisability theory analysis modelling both item and occasion facets . For that study's 120‐item test, we computed reliability coefficients based on the estimated variance components that were reported; the test–retest reliability computed from the generalisability study output was only r = 0.45, and the overall generalisability coefficient for the 120‐item test, administered twice, was only 0.40.…”

Section: Reliability and Internal Structure Of Sctsmentioning

confidence: 99%

See 1 more Smart Citation

Threats to validity in the use and interpretation of script concordance test scores

2013

View full text Add to dashboard Cite

Context Recent reviews have claimed that the script concordance test (SCT) methodology generally produces reliable and valid assessments of clinical reasoning and that the SCT may soon be suitable for high‐stakes testing. Objectives This study is intended to describe three major threats to the validity of the SCT not yet considered in prior research and to illustrate the severity of these threats. Methods We conducted a review of SCT reports available through the Web of Science database. Additionally, we reanalysed scores from a previously published SCT administration to explore issues related to standard SCT scoring practice. Results Firstly, the predominant method for aggregate and partial credit scoring of SCTs introduces logical inconsistencies in the scoring key. Secondly, our literature review shows that SCT reliability studies have generally ignored inter‐panel, inter‐panellist and test–retest measurement error. Instead, studies have focused on observed levels of coefficient alpha, which is neither an informative index of internal structure nor a comprehensive index of reliability for SCT scores. As such, claims that SCT scores show acceptable reliability are premature. Finally, SCT criteria for item inclusion, in concert with a statistical artefact of the SCT format, cause anchors at the extremes of the scale to have less expected credit than anchors near or at the midpoint. Consequently, SCT scores are likely to reflect construct‐irrelevant differences in examinees' response styles. This makes the test susceptible to bias against candidates who endorse extreme scale anchors more readily; it also makes two construct‐irrelevant test taking strategies extremely effective. In our reanalysis, we found that examinees could drastically increase their scores by never endorsing extreme scale points. Furthermore, examinees who simply endorsed the scale midpoint for every item would still have outperformed most examinees who used the scale as it is intended. Conclusions Given the severity of these threats, we conclude that aggregate scoring of SCTs cannot be recommended. Recommendations for revisions of SCT methodology are discussed.

show abstract

Section: Reliability and Internal Structure Of Sctsmentioning

confidence: 54%

Section: Reliability and Internal Structure Of Sctsmentioning

confidence: 99%

Threats to validity in the use and interpretation of script concordance test scores

2013

View full text Add to dashboard Cite

show abstract

“…Klinisch redeneren vraagt veel oefening met vaak onvolledige informatie in een authentieke casus. 20 Dit denkproces draagt niet een generaliserend maar een individualiserend karakter. 21 Dat wil zeggen dat er geen kant en klare gestandaardiseerde en geprotocolleerde oplossingen beschikbaar zijn; ieder (dreigend) gezondheidsprobleem in de gegeven context vraagt om een weloverwogen klinische redenering om tot het juiste handelen te kunnen komen.…”

Section: Het 4 C/id Modelunclassified

Klinisch redeneren in het fysiotherapie onderwijs

Gerards-Last

Geraets

2011

TS MEDISCH ONDERWIJS

View full text Add to dashboard Cite

show abstract

“…The SCT has been applied in several target groups [6, 9–11], among which pre-clinical medical students [12, 13]. At the University Medical Center Utrecht, the SCT was applied among second-year medical students, as a final test of a course in case-based clinical reasoning in 2009–2010.…”

Section: Introductionmentioning

confidence: 99%

Effects of two different instructional formats on scores and reliability of a script concordance test

et al. 2012

View full text Add to dashboard Cite

The script concordance test (SCT) is designed to assess clinical reasoning by adapting the likelihood of a case diagnosis, based on provided new information. In the standard instructions students are asked to exclude alternative diagnoses they have in mind when answering the questions, but it might be more authentic to include these. Fifty-nine final-year medical students completed an SCT. Twenty-nine were asked to take their differential diagnosis into account (adapted instructions). Thirty students were asked not to consider other diagnoses (standard instructions). All participants were asked to indicate for each question whether they were confused answering it with the given instructions (‘confusion indication’). Mean score of the test with the adapted instructions was 81.5 (SD 3.8) and of the test with the standard instructions 82.9 (SD 5.0) (p = 0.220). Cronbach’s alpha was 0.39 for the adapted instructions and 0.66 for the standard instructions. The mean number of confusion indications was 4.2 (SD 4.4) per student for the adapted instructions and 16.7 (SD 28.5) for the standard instructions (p = 0.139). Our attempt to improve SCTs reliability by modifying the instructions did not lead to a higher alpha; therefore we do not recommend this change in the instructional format.

show abstract

Assessment of competence in clinical reasoning and decision‐making under uncertainty: the script concordance test method

Cited by 20 publications

References 30 publications

Threats to validity in the use and interpretation of script concordance test scores

Threats to validity in the use and interpretation of script concordance test scores

Klinisch redeneren in het fysiotherapie onderwijs

Effects of two different instructional formats on scores and reliability of a script concordance test

Contact Info

Product

Resources

About