Setting defensible standards in small cohort OSCEs: Understanding better when borderline regression can ‘work’

Homer, Matt; Fuller, Richard; Hallam, Jennifer; Pell, Godfrey

doi:10.1080/0142159x.2019.1681388

Cited by 16 publications

(13 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, the BGM can have a potential problem when the borderline group is not sufficient, such as in Station 5 of this study. The same problem was also found in the study of Wood et al [13]. In their study, the borderline group was 20% (12/59 examinees), and the difference in the pass rate was 69% in the BGM and 92% in the BRM.…”

Section: Comparison With Previous Studiessupporting

confidence: 81%

“…If there are insufficient cohorts evaluated as borderline, cut scores may be calculated based on a relatively small number of examinees, which may increase the statistical error associated with the cut score [12]. As the score distribution is left-skewed and the borderline group is at the lower thin tail of the overall score distribution, the mean or median will be biased toward the high side [13].…”

Section: Discussionmentioning

confidence: 99%

“…However, OSCEs in many medical schools may have small cohorts, such as a singleyear group. Homer et al [12] have shown that the use of the BRM in the context of small cohorts can be generally successful. They investigated the use of the BRM in different high stakes assessment contexts and found that the BRM functions effectively at most stations.…”

Section: Comparison With Previous Studiesmentioning

confidence: 99%

See 2 more Smart Citations

Comparing the cut score for the borderline group method and borderline regression method with norm-referenced standard setting in an objective structured clinical examination in medical school in Korea

Park

Lee

Kim

et al. 2021

J Educ Eval Health Prof

View full text Add to dashboard Cite

Purpose: Setting standards is critical in health professions. However, appropriate standard setting methods do not always apply to the set cut score in performance assessment. The aim of this study was to compare the cut score when the standard setting is changed from the norm-referenced method to the borderline group method (BGM) and borderline regression method (BRM) in an objective structured clinical examination (OSCE) in medical school.Methods: This was an explorative study to model of the BGM and BRM. A total of 107 fourth-year medical students attended the OSCE at seven stations with encountering standardized patients (SPs) and one station with performing skills on a manikin on 15 July 2021. Thirty-two physician examiners evaluated the performance by completing a checklist and global rating scales.Results: The cut score of the norm-referenced method was lower than that of the BGM (p<0.01) and BRM (p<0.02). There was no significant difference in the cut score between the BGM and BRM (p=0.40). The station with the highest standard deviation and the highest proportion of the borderline group showed the largest cut score difference in standard setting methods.Conclusion: Prefixed cut scores by the norm-referenced method without considering station contents or examinee performance can vary due to station difficulty and content, affecting the appropriateness of standard setting decisions. If there is an adequate consensus on the criteria for the borderline group, standard setting with the BRM could be applied as a practical and defensible method to determine the cut score for OSCE.

show abstract

Section: Comparison With Previous Studiessupporting

confidence: 81%

Section: Discussionmentioning

confidence: 99%

Section: Comparison With Previous Studiesmentioning

confidence: 99%

See 1 more Smart Citation

Comparing the cut score for the borderline group method and borderline regression method with norm-referenced standard setting in an objective structured clinical examination in medical school in Korea

Park

Lee

Kim

et al. 2021

J Educ Eval Health Prof

View full text Add to dashboard Cite

show abstract

“…Separate analysis not included here indicates that the typical internal consistency reliability of the examination is on average relatively high (mean alpha = 0.76 for station scores across the 349 exams), and BRM has been shown to work well in this setting (Homer et al 2019). Candidate level data was not available for analysis, an issue we will return to at relevant points in the paper.…”

Section: Data Samplementioning

confidence: 99%

Re-conceptualising and accounting for examiner (cut-score) stringency in a ‘high frequency, small cohort’ performance test

Homer

2020

Adv in Health Sci Educ

Self Cite

View full text Add to dashboard Cite

Variation in examiner stringency is an ongoing problem in many performance settings such as in OSCEs, and usually is conceptualised and measured based on scores/grades examiners award. Under borderline regression, the standard within a station is set using checklist/ domain scores and global grades acting in combination. This complexity requires a more nuanced view of what stringency might mean when considering sources of variation of cut-scores in stations. This study uses data from 349 administrations of an 18-station, 36 candidate single circuit OSCE for international medical graduates wanting to practice in the UK (PLAB2). The station-level data was gathered over a 34-month period up to July 2019. Linear mixed models are used to estimate and then separate out examiner (n = 547), station (n = 330) and examination (n = 349) effects on borderline regression cut-scores. Examiners are the largest source of variation in cut-scores accounting for 56% of variance in cut-scores, compared to 6% for stations, < 1% for exam and 37% residual. Aggregating to the exam level tends to ameliorate this effect. For 96% of examinations, a 'fair' cutscore, equalising out variation in examiner stringency that candidates experience, is within one standard error of measurement (SEM) of the actual cut-score. The addition of the SEM to produce the final pass mark generally ensures the public is protected from almost all false positives in the examination caused by examiner cut-score stringency acting in candidates' favour.

show abstract

“…Many institutions are exploring the use of technology enhanced assessment solutions for online knowledge testing, and delivery of 'online OSCEs', which involved structured oral examination components. A recent paper has reported successful re-design and delivery of a face to face OSCE whilst adhering to strict infection prevention restrictions (Boursicot et al 2020), successfully using borderline method regression methods to standard set for small cohorts (Homer et al 2016). However, these options may not be feasible for institutions with limited resources.…”

Section: Theme 3: Selection and Assessmentmentioning

confidence: 99%

Adapting to the impact of COVID-19: Sharing stories, sharing practice

et al. 2020

Self Cite

View full text Add to dashboard Cite

Health Professions' Educators (HPEs) and their learners have to adapt their educational provision to rapidly changing and uncertain circumstances linked to the COVID-19 pandemic. This paper reports on an AMEE-hosted webinar: Adapting to the impact of COVID-19: Sharing stories, sharing practice. Attended by over 500 colleagues from five continents, this webinar focused on the impact of the virus across the continuum of education and training. Short formal presentations on teaching and learning, assessment, selection and postgraduate training generated wide-ranging questions via the Chatbox. A thematic analysis of the Chatbox thread indicated the most pressing concerns and challenges educators were experiencing in having to adapt programmes and learning across the continuum of medical education and training. The main areas of concern were: campus-based teaching and learning; clinical teaching; selection and assessment, and educator needs. While there is clearly no one simple solution to the unprecedented issues medical education and training face currently, there were two over-arching messages. First, this is a time for colleagues across the globe to help and support each other. Second, many local responses and innovations could have the potential to change the shape of medical education and training in the future.

show abstract

Setting defensible standards in small cohort OSCEs: Understanding better when borderline regression can ‘work’

Abstract: This is a repository copy of Setting defensible standards in small cohort OSCEs: Understanding better when borderline regression can 'work'.

Cited by 16 publications

References 34 publications

Comparing the cut score for the borderline group method and borderline regression method with norm-referenced standard setting in an objective structured clinical examination in medical school in Korea

Comparing the cut score for the borderline group method and borderline regression method with norm-referenced standard setting in an objective structured clinical examination in medical school in Korea

Re-conceptualising and accounting for examiner (cut-score) stringency in a ‘high frequency, small cohort’ performance test

Adapting to the impact of COVID-19: Sharing stories, sharing practice

Contact Info

Product

Resources

About