The Relationship Between Licensing Examination Performance and the Outcomes of Care by International Medical School Graduates

Norcini, John J.; Boulet, John R.; Opalek, Amy; Dauphinee, W. Dale

doi:10.1097/acm.0000000000000310

Cited by 106 publications

(88 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Performance in these examinations predicts performance in clinical practice and other medical assessment processes11 18, 19. For example, each part of the MRCP examination predicts the next17, and candidates who score higher in all parts of the MRCP examination do better in workplace‐based assessments than those who underperform16.…”

Section: Discussionmentioning

confidence: 99%

Impact of performance in a mandatory postgraduate surgical examination on selection into specialty training

et al. 2017

View full text Add to dashboard Cite

BackgroundThe Intercollegiate Membership of the Royal College of Surgeons (MRCS) examination is undertaken by large numbers of trainees in the UK and internationally as a mandatory step within surgical training. Unlike some high‐stakes medical examinations, the MRCS is yet to be validated. A quantitative study was undertaken to assess its predictive validity by investigating the relationship between MRCS (Parts A and B) and national selection interview scores for general and vascular surgery in the UK.MethodsPearson correlation coefficients were used to examine the linear relationship between each assessment, and linear regression analyses were employed to identify potential independent predictors of the national selection score. All UK medical graduates who attempted the interview in 2011–2015 were included.ResultsSome 84·4 per cent of the candidates (1231 of 1458) were matched with MRCS data. There was a significant positive correlation between the first attempt score at Part B of the MRCS examination and the national selection score (r = 0·38, P < 0·001). In multivariable analysis, 17 per cent of variance in the national selection first attempt score was explained by the Part B MRCS score and number of attempts (change in R 2 value of 0·10 and 0·07 respectively; P < 0·001). Candidates who required more than two attempts at Part B were predicted to score 8·1 per cent less than equally matched candidates who passed at their first attempt.ConclusionThis study supports validity of the MRCS examination, and indicates its predictive value regarding entry into specialist training.

show abstract

Section: Discussionmentioning

confidence: 99%

Impact of performance in a mandatory postgraduate surgical examination on selection into specialty training

et al. 2017

View full text Add to dashboard Cite

show abstract

“…Since then researchers have tried to establish such a link. Both Norcini et al (2014) and Tamblyn et al (2007) found correlations between licensing examination performance (in the USA and Canada respectively) and subsequent specific patient outcomes [38], or rates of complaints [39]. However, as discussed in more detail below, this evidence for a correlation with other variables is not supported by evidence for better patient outcomes as a consequence of NLEs.…”

Section: Resultsmentioning

confidence: 99%

“…Although the aforementioned studies by Norcini et al [38] and Tamblyn et al [39] demonstrate excellent arguments for the importance of testing, and medical education more generally, their findings are limited to establishing correlations between testing and outcomes and not causation. In other words, there is evidence that better doctors do better in NLEs, but not that doctors improve as a consequence of introducing NLEs; this kind of before and after evidence is absent in the extant literature.…”

Section: Resultsmentioning

confidence: 99%

The impact of large scale licensing examinations in highly developed countries: a systematic review

et al. 2016

View full text Add to dashboard Cite

BackgroundTo investigate the existing evidence base for the validity of large-scale licensing examinations including their impact.MethodsSystematic review against a validity framework exploring: Embase (Ovid Medline); Medline (EBSCO); PubMed; Wiley Online; ScienceDirect; and PsychINFO from 2005 to April 2015. All papers were included when they discussed national or large regional (State level) examinations for clinical professionals, linked to examinations in early careers or near the point of graduation, and where success was required to subsequently be able to practice. Using a standardized data extraction form, two independent reviewers extracted study characteristics, with the rest of the team resolving any disagreement. A validity framework was used as developed by the American Educational Research Association, American Psychological Association, and National Council on Measurement in Education to evaluate each paper’s evidence to support or refute the validity of national licensing examinations.Results24 published articles provided evidence of validity across the five domains of the validity framework. Most papers (n = 22) provided evidence of national licensing examinations relationships to other variables and their consequential validity. Overall there was evidence that those who do well on earlier or on subsequent examinations also do well on national testing. There is a correlation between NLE performance and some patient outcomes and rates of complaints, but no causal evidence has been established.ConclusionsThe debate around licensure examinations is strong on opinion but weak on validity evidence. This is especially true of the wider claims that licensure examinations improve patient safety and practitioner competence.

show abstract

“…This will require an emphasis on a robust, programmatic approach to assessment that ideally focuses on workplace-based formative assessment, rather than isolated high-stakes point in time-summative examinations. This is not to imply that there is not a role for high-stakes examination, as it can be useful to predict future patient outcomes [24], but rather that the opportunity for lower-stakes, more frequent assessment may be invaluable for learning on an ongoing basis [51]. Moving up to Miller's top level of "Does" (Fig.…”

Section: Shifting the Assessment Paradigm For Competency-based Educationmentioning

confidence: 98%

Simulation Curriculum Development, Competency-Based Education, and Continuing Professional Development

Pirie

Kappus

Sudikoff

et al. 2016

Comprehensive Healthcare Simulation: Pediatrics

View full text Add to dashboard Cite

Simulation Pearls• There are increasing calls for healthcare professionals to fulfill their social contract with society and ensure competence of all professionals in order to maintain the privilege of self-regulation. Competency-based education (CBE) offers promise, as an outcome-based model of education, to help address the gap between actual and desired performance.• Simulation-based education (SBE) curricula should be based on needs analysis. Prior to designing, clear goals should be defined to measure the success of the training program.• Specific learning objectives, instructional strategy, simulation technology, training environment, and debriefing models should be carefully selected based on the level of learner.• Challenges to CBE include defining learning objectives that are not excessively comprehensive, trainees focusing on milestones rather than achieving excellence, administrative logistics, instructor expertise and availability, and cost.• Optimal education will require a change in our current approach to assessment. Assessment needs to be programmatic and conceptualized as part of instructional design with a shift away from assessment of learning to assessment for learning.• SBE is increasingly used for high-stakes, summative purposes such as local program-based examinations, achieving certification, and demonstrating ongoing competence to maintain certification.

show abstract

The Relationship Between Licensing Examination Performance and the Outcomes of Care by International Medical School Graduates

Abstract: These findings provide evidence for the validity of Step 2 CK scores. Given the magnitude of its relationship with patient outcomes, the results support the use of the examination as an effective screening strategy for licensure.

Cited by 106 publications

References 14 publications

Impact of performance in a mandatory postgraduate surgical examination on selection into specialty training

Impact of performance in a mandatory postgraduate surgical examination on selection into specialty training

The impact of large scale licensing examinations in highly developed countries: a systematic review

Simulation Curriculum Development, Competency-Based Education, and Continuing Professional Development

Contact Info

Product

Resources

About