Unpacking the dark variance of differential attainment on examinations in overseas graduates

Patterson, Fiona; Tiffin, Paul A.; Lopes, Safiatu; Zibarras, Lara

doi:10.1111/medu.13605

Cited by 31 publications

(52 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A total of 10 were cross-sectional studies, [27][28][29][30][31][32][33][34][35]55 where the outcome was measured at the same time or in the same selection cycle as taking the SJT. A total of 17 were cohort studies 17,[36][37][38][39][40][41][42][43][44][45][46][47][48][49][50][51] that had a follow-up period before the outcome of interest was measured. Three studies employed a mixture of F I G U R E 1 PRISMA (preferred reporting items for systematic reviews and meta-analyses) flowchart for the systematic review Abbreviations: CINAHL, Cumulative Index to Nursing and Allied Health Literature; EMBASE, Excerpta Medica Database; ERIC, Educational Resources Information Center; EThos, Electronic Theses Online Service; UCAT, University Clinical Aptitude Test cross-sectional and more distal outcomes.…”

Section: Re Sultsmentioning

confidence: 99%

“…[52][53][54] The length of followup across the cohort studies varied from 1 to 9 years after taking the SJT. Full details of the included studies are listed in Table S1. A total of 11 studies 17,28,29,35,[40][41][42][43]47,49,50 looked at undergraduate selection for medical school entry, five studies 31,36,37,45,51 at entry to Foundation Year training programmes (the first 2 years of post-qualification training in the United Kingdom [UK]) and 14 studies 27,30,[32][33][34]38,39,44,46,48,[52][53][54][55] at entry to specialty training. The youngest participant mean age was 17.9 years and the oldest was 34.0 years.…”

Section: Re Sultsmentioning

confidence: 99%

See 1 more Smart Citation

Situational judgement test validity for selection: A systematic review and meta‐analysis

et al. 2020

Self Cite

View full text Add to dashboard Cite

Context Situational judgement tests (SJTs) are widely used to evaluate ‘non‐academic’ abilities in medical applicants. However, there is a lack of understanding of how their predictive validity may vary across contexts. We conducted a systematic review and meta‐analysis to synthesise existing evidence relating to the validity of such tools for predicting outcomes relevant to interpersonal workplace performance. Methods Searches were conducted in relevant databases to June 2019. Study quality and risk of bias were assessed using the Quality In Prognosis Studies (QUIPS) tool. Results were pooled using random effects meta‐analysis and meta‐regressions. Results Initially, 470 articles were identified, 218 title or abstracts were reviewed, and 44 full text articles were assessed with 30 studies meeting the final inclusion criteria and were judged, overall, to be at moderate risk of bias. Of these, 26 reported correlation coefficients relating to validity, with a pooled estimate of 0.32 (95% confidence interval 0.26 to 0.39, P < .0001). Considerable heterogeneity was observed (I2 = 96.5%) with the largest validity coefficients tending to be observed for postgraduate, rather than undergraduate, selection studies (β = 0.23, 0.11 to 0.36, P < .001). The correction of validity coefficients for attenuation was also independently associated with larger effects (β = 0.13, 0.03 to 0.23, P = .01). No significant associations with test medium (video vs text format), cross‐sectional study design, or period of assessment (one‐off vs longer‐term) were observed. Where reported, the scores generally demonstrated incremental predictive validity, over and above tests of knowledge and cognitive ability. Conclusions The use of SJTs in medical selection is supported by the evidence. The observed trend relating to training stage requires investigation. Further research should focus on developing robust criterion‐relevant outcome measures that, ideally, capture interpersonal aspects of typical workplace performance. This will facilitate additional work identifying the optimal place of SJTs within particular selection contexts and further enhancing their effectiveness.

show abstract

Section: Re Sultsmentioning

confidence: 99%

Section: Re Sultsmentioning

confidence: 99%

Situational judgement test validity for selection: A systematic review and meta‐analysis

et al. 2020

Self Cite

View full text Add to dashboard Cite

show abstract

“…The differences across Chinese and US Caucasians may be because English was a second language for all our Chinese respondents even though they passed TOEFL requirements to enter the university. The important role of language is reflected in a paper by Patterson, Tiffin, Lopes, and Zibarras (). In that paper, the authors found that English language skills were related to a test of practical clinical skills in a medical admissions context and that this relationship was primarily mediated by scores on an SJT.…”

Section: Discussionmentioning

confidence: 99%

Culture as a determinant of option choice in a situational judgement test: A new look

Schmitt

Prasad

Ryan

et al. 2018

J Occupat & Organ Psyc

View full text Add to dashboard Cite

Past research on cultural correlates of biased items in multiple‐choice tests have almost always considered post‐hoc explanations that have not been very convincing and they have considered only differential subgroup responses to the correct or preferred answer. In an effort to determine the degree to which culture explains the difference between Chinese and Americans’ responses to a situational judgement test (SJT), 135 options to 25 SJT items were rated on the degree to which they reflected seven dimensions of culture. Ratings of the cultural content of the options were significantly and moderately correlated with Chinese–American differences in Most Likely responses to the options. In addition, a loglinear analysis of differential distractor functioning (DDF) yielded results consistent with the hypothesis that culture was at least partly the reason for subgroup differences in option choices and the resultant findings of DDF. The results suggest that SJTs and perhaps other non‐academic measures must be used with caution across cultural groups. Practitioner points Consultation with experts and test takers in the cultures with which you hope to use an instrument should always precede test use. Review with them both the item stems and options (if any) for items that may have a different meaning across cultural groups. Measurement invariance of instruments used with different cultural groups should be assessed. When sample sizes do not permit sophisticated analyses such as confirmatory factor analyses or item response theory analyses, examine and seek to understand any simple descriptive statistics that reflect a difference across cultural groups. There should be an examination of examine response differences to distractors in multiple‐choice items as well as differences in responses to correct options.

show abstract

“…Moreover, a previous review of the use of SJTs for the evaluation of non‐academic traits in general concluded that there was evidence of cost‐effectiveness compared to other approaches (eg interviews) . Emerging evidence also indicates that, to date, in a variety of undergraduate and postgraduate medical selection settings, at least modest correlations are observed between performance on these traditional SJTs and subsequent relevant outcomes that reflect aspects of social functioning . This suggests, at least in this context, such selection SJTs are generally estimating, to some extent, knowledge of interpersonal functioning.…”

Section: A Brief History Of Sjts For Personnel Selectionmentioning

confidence: 99%

“…Indeed, there may be little relationship between performance on such instruments and academic achievement in the early years of medical undergraduate education . Evidence for the validity of SJTs in evaluating non‐academic traits for medical selection purposes has also been sought by linking the test scores to outcomes that may require a degree of interpersonal competence, such as performance in high‐fidelity simulations of clinical practice in primary care physicians, successful completion of the first stage of postgraduate medical training or performance in the first year as a general practitioner . Cross‐sectional evidence that supports the concept of SJTs measuring constructs relevant to interpersonal functioning also exists, in the form of correlations with multiple mini‐interviews .…”

Section: Traditional Vs Construct‐driven Sjts: Psychometric Propertiesmentioning

confidence: 99%

Situational judgement tests for selection: Traditional vs construct‐driven approaches

et al. 2019

Self Cite

View full text Add to dashboard Cite

A situational judgement test (SJT) is an assessment whereby a candidate is presented with a specific scenario and must evaluate several possible responses to the scenario. The response format can vary but commonly involves ranking potential behavioural responses in order of either appropriateness or perceived effectiveness. Another commonly employed choice format involves a candidate choosing the 'best' and 'worst' behaviours depicted. An example of a Abstract Context: Historically, situational judgement tests (SJTs) have been widely used for personnel selection. Their use in medical selection in Europe is growing, with plans for further expansion into North America and Australasia, in an attempt to measure and select on 'non-academic' personal attributes. However, there is a lack of clarity regarding what such tests actually measure and how they should be designed, scored and implemented within the medical and health education selection process. In particular, the theoretical basis from which such tests are developed will determine the scoring options available, influencing their psychometric properties and, ultimately, their validity. Methods:The aim of this article is to create an awareness of the previous theory and practice that has informed SJT development. We describe the emerging interest in the use of the SJT format to measure specific constructs (eg 'resilience', 'dependability', etc.), drawing on the tradition of 'individual differences' psychology. We compare and contrast this newer 'construct-driven' method with the traditional, pragmatic approach to SJT creation, often employed by organisational psychologists. Making reference to measurement theory, we highlight how the anticipated psychometric properties of traditional vs construct-driven SJTs are likely to differ.Conclusions: Compared to traditional SJTs, construct-driven SJTs have a strong theoretical basis, are uni-rather than multidimensional, and may behave more like personality self-report instruments. Emerging evidence also suggests that construct-driven SJTs have comparable predictive validity for workplace performance, although they may be more prone to 'faking' effects. It is possible that construct-driven approaches prove more appropriate at early stages of medical selection, where candidates have little or no health care work experience. Conversely, traditional SJTs may be more suitable for specialty recruitment, where a range of hypothetical workplace scenarios can be sampled in assessments. S U PP O RTI N G I N FO R M ATI O N Additional supporting information may be found online in the Supporting Information section. How to cite this article: Tiffin PA, Paton LW, O'Mara D, MacCann C, Lang JWB, Lievens F. Situational judgement tests for selection: Traditional vs construct-driven approaches. Med Educ. 2020;54:105-115.

show abstract

Unpacking the dark variance of differential attainment on examinations in overseas graduates

Cited by 31 publications

References 30 publications

Situational judgement test validity for selection: A systematic review and meta‐analysis

Situational judgement test validity for selection: A systematic review and meta‐analysis

Culture as a determinant of option choice in a situational judgement test: A new look

Situational judgement tests for selection: Traditional vs construct‐driven approaches

Contact Info

Product

Resources

About