Initial Validity Analysis of the American Board of Emergency Medicine Enhanced Oral Examination

Kowalenko, Terry; Heller, Barry; Strauss, Robert W.; Counselman, Francis L.; Mallory, Mary Nan; Joldersma, Kevin B.; Coombs, Andrea B.; Harvey, Anne L.; Reisdorff, Earl J.

doi:10.1111/acem.13068

Cited by 5 publications

(13 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Prior studies of the in-person OCE showed substantial validity and reliability support. [4][5][6][7][8] To maintain a similarly high level of validity and reliability, ABEM designed the VOE to be similar to the OCE in content and administration. The OCE had amassed substantial validity evidence as previously reported.…”

Section: Discussionmentioning

confidence: 99%

“…This study is the first report of validity and reliability evidence for the virtual format of the ABEM certifying examination. Prior studies of the in‐person OCE showed substantial validity and reliability support 4–8 . To maintain a similarly high level of validity and reliability, ABEM designed the VOE to be similar to the OCE in content and administration.…”

Section: Discussionmentioning

confidence: 99%

“…The two post‐examination survey items of interest that assessed content relevance applied a 5‐point Likert scale (strongly disagree, disagree, neutral, agree, and strongly agree). The items were: (1) “Overall, the types of cases on this examination were cases that an emergency physician should be expected to see”; and (2) “In my practice, I have seen most of these cases.” These survey items were developed by an expert panel of clinically active emergency medicine experts and have been used to support validity claims on prior OCEs 5 …”

Section: Methodsmentioning

confidence: 99%

“…Given the high‐stakes nature of specialty board certification, it was important to determine the degree to which the VOE was reliable as well as determine the presence of validity evidence to support the new format. The in‐person OCE had substantial validity evidence and reliability data to support its use in the certification of emergency physicians 4–8 . The purpose of this investigation was to determine whether there is sufficient early validity and reliability evidence to support the continued use of the ABEM VOE for certification decisions.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Early validity and reliability evidence for the American Board of Emergency Medicine Virtual Oral Examination

Chudnofsky

Reisdorff

Joldersma

et al. 2023

AEM Education and Training

Self Cite

View full text Add to dashboard Cite

Background The American Board of Emergency Medicine (ABEM) in‐person Oral Certification Examination (OCE) was halted abruptly in 2020 due to the COVID‐19 pandemic. The OCE was reconfigured to be administered in a virtual environment starting in December 2020. Objectives The purpose of this investigation was to determine whether there was sufficient validity and reliability evidence to support the continued use of the ABEM virtual Oral Examination (VOE) for certification decisions. Methods This retrospective, descriptive study used multiple data sources to provide validity evidence and reliability data. Validity evidence focused on test content, response processes, internal structure (e.g., internal consistency and item response theory), and the consequences of testing. A multifaceted Rasch reliability coefficient was used to measure reliability. Study data were from two 2019 in‐person OCEs and the first four VOE administrations. Results There were 2279 physicians who took the 2019 in‐person OCE examination and 2153 physicians who took the VOE during the study period. Among the OCE group, 92.0% agreed or strongly agreed that the cases on the examination were cases that an emergency physician should be expected to see; 91.1% of the VOE group agreed or strongly agreed. A similar pattern of responses given to a question about whether the cases on the examination were cases that they had seen. Additional evidence of validity was obtained by the use of the EM Model, the process for case development, the use of think‐aloud protocols, and similar test performance patterns (e.g., pass rates). For reliability, the Rasch reliability coefficients for the OCE and the VOE during the study period were all >0.90. Conclusions There was substantial validity evidence and reliability to support ongoing use of the ABEM VOE to make confident and defensible certification decisions.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Early validity and reliability evidence for the American Board of Emergency Medicine Virtual Oral Examination

Chudnofsky

Reisdorff

Joldersma

et al. 2023

AEM Education and Training

Self Cite

View full text Add to dashboard Cite

show abstract

“…The examination has been studied thoroughly and has been found to have sufficient validity evidence and reliability support to make summative certification decisions. [13][14][15][16][17][18] During the COVID-19 pandemic, ABEM converted the administration of the OCE to a virtual platform. OCE results from the virtual oral examination format are excluded from this investigation.…”

Section: The Bottom Linementioning

confidence: 99%

Comparison of performance data between emergency medicine 1‐3 and 1‐4 program formats

et al. 2023

Self Cite

View full text Add to dashboard Cite

show abstract

Association between the American Board of Emergency Medicine Oral Certifying Examination and Future State Medical Board Disciplinary Actions

Reisdorff,

Johnston,

Kraus

et al. 2024

JACEP Open

Self Cite

View full text Add to dashboard Cite

Objectives:The American Board of Emergency Medicine (ABEM) requires a written examination (the Qualifying Examination) followed by the Oral Certifying Examination (OCE) to obtain ABEM certification. Maintaining ABEM certification is associated with fewer state medical board (SMB) disciplinary actions. We sought to determine the association between poor initial performance on the OCE and subsequent severe SMB disciplinary action.MethodsWe included physicians who completed US categorical emergency medicine residencies in 2016 and earlier. We classified OCE performance as good (passed on first attempt) and poor (never passed or required > 1 attempt to pass). We obtained data on physician SMB disciplinary actions from the National Practitioner Data Bank that were limited to actions that denied licensure or altered the status of a medical license (eg, suspension). We determined the association between poor OCE performance and subsequent severe SMB disciplinary action.ResultsOf 34,871, 93.5% passed the OCE on the first attempt, 6.1% required multiple attempts, and 0.3% never passed. Of the physicians (93.5%) with good OCE performance, 1.0% received a severe SMB action. Among physicians with poor OCE performance, 2.3% received a severe action; and of those who never passed, 1.7% received a severe action (Table 1). Poor OCE performance was associated with an increased odds of severe SMB disciplinary action (OR 2.21, 95% CI: 1.57–3.12).ConclusionPhysicians with poor OCE performance exhibited higher odds of experiencing a subsequent severe SMB disciplinary action. The OCE may have utility as a predictor of future professionalism or clinical performance.

show abstract

Initial Validity Analysis of the American Board of Emergency Medicine Enhanced Oral Examination

Cited by 5 publications

References 8 publications

Early validity and reliability evidence for the American Board of Emergency Medicine Virtual Oral Examination

Early validity and reliability evidence for the American Board of Emergency Medicine Virtual Oral Examination

Comparison of performance data between emergency medicine 1‐3 and 1‐4 program formats

Association between the American Board of Emergency Medicine Oral Certifying Examination and Future State Medical Board Disciplinary Actions

Contact Info

Product

Resources

About