Threats to validity in the use and interpretation of script concordance test scores

Abstract

Context Recent reviews have claimed that the script concordance test (SCT) methodology generally produces reliable and valid assessments of clinical reasoning and that the SCT may soon be suitable for high‐stakes testing. Objectives This study is intended to describe three major threats to the validity of the SCT not yet considered in prior research and to illustrate the severity of these threats. Methods We conducted a review of SCT reports available through the Web of Science database. Additionally, we reanalysed scores from a previously published SCT administration to explore issues related to standard SCT scoring practice. Results Firstly, the predominant method for aggregate and partial credit scoring of SCTs introduces logical inconsistencies in the scoring key. Secondly, our literature review shows that SCT reliability studies have generally ignored inter‐panel, inter‐panellist and test–retest measurement error. Instead, studies have focused on observed levels of coefficient alpha, which is neither an informative index of internal structure nor a comprehensive index of reliability for SCT scores. As such, claims that SCT scores show acceptable reliability are premature. Finally, SCT criteria for item inclusion, in concert with a statistical artefact of the SCT format, cause anchors at the extremes of the scale to have less expected credit than anchors near or at the midpoint. Consequently, SCT scores are likely to reflect construct‐irrelevant differences in examinees' response styles. This makes the test susceptible to bias against candidates who endorse extreme scale anchors more readily; it also makes two construct‐irrelevant test taking strategies extremely effective. In our reanalysis, we found that examinees could drastically increase their scores by never endorsing extreme scale points. Furthermore, examinees who simply endorsed the scale midpoint for every item would still have outperformed most examinees who used the scale as it is intended. Conclusions Given the severity of these threats, we conclude that aggregate scoring of SCTs cannot be recommended. Recommendations for revisions of SCT methodology are discussed.

Script concordance test in medical schools in Brazil: possibilities and limitations

“…4 Therefore, it may be inferred that the more senior students will interpret the data in the scenarios presented and make decisions with a higher degree of agreement with the reference panel. 4,16 The SCT was able to demonstrate that students in the preclinical and clinical phases had different levels of performance according to the degree of knowledge and maturity expected at each stage. 10,16 Problems with SCT construction regarding several questions were identified in the present study.…”

“…2,4,6 In smaller academic units with limited resources, it may be more efficient to replace the aggregate five-point scoring methodology 6 with consensus scoring methodology using a three-point scale ("unlikely", "neither likely nor unlikely" and "likely"), thereby possibly avoiding contradictory values in the scoring key. 4 In medical education practice, all assessment instruments have limitations. 13 The SCT was developed within the context of the cognitive psychology approach known as script theory in order to assess how students and doctors organize their knowledge.…”

“…When panelists disagree in such a fundamental manner, it may indicate that there were problems in the construction of the question and that the right answer cannot be known. 4 The SCT instrument in this study had a reliability index (Cronbach's alpha) 17 of 0.64, for an expected value of 0.70 -0.90. 16 Some studies have reported similar results, in which this index is less than desirable.…”

“…3 The SCT is intended to evaluate the information interpretation process. 4 The instrument has been developed in various educational settings and in different countries and languages and is based on written presentation of short clinical cases, followed by a choice of diagnostic and therapeutic decisions. 1,[5][6][7][8][9][10][11][12][13] Routine use of the SCT is not common in Brazilian universities.…”

Scoring method of a Situational Judgment Test: influence on internal consistency reliability, adverse impact and correlation with personality?

“…Narrow Likert scales may not be as strongly affected by response tendencies as Likert scales with more scale points (Flaskerud 1988), resulting in no differences when controlling for the response tendencies. A study on script concordance tests recommended a reduction of the Likert scale from five to three points in order to decrease the influence of construct-irrelevant factors such as examinee response styles (Lineberry et al 2013). Dichotomizing the Likert scale does seem to have some effect on adverse impact, but at the cost of low internal consistency reliability, leading to a similar issue as the diversity-validity dilemma (De Soete et al 2013).…”