Standard Setting Methods for Pass/Fail Decisions on High-Stakes Objective Structured Clinical Examinations: A Validity Study

Yousuf, Naveed; Violato, Claudio; Zuberi, Rukhsana Wamiq

doi:10.1080/10401334.2015.1044749

Cited by 26 publications

(13 citation statements)

References 51 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…For adjustment of cut-scores from retrospective standard setting, stations item scores and GRS scores are used in a formula for Borderline Regression Analysis 36 ; 37 . Discrepancies between station item scores and GRS results affect the outcome of Borderline Regression Analysis and R 2 , and Intergrade discriminators as the slope of the regression curve are affected due to high variation in scores on both scales 37,38 . We think that the latter has serious implications for the validity of our OSCEs.…”

Section: Discussionmentioning

confidence: 99%

Cross-institutional OSCE Quality Assurance in Europe using a mutual assessment strategy; are we equipped for it?

Kropmans¹,

Søfteland²,

Wijnalda³

et al. 2020

Preprint

View full text Add to dashboard Cite

BackgroundUntil 2008, Objective Structured Clinical Examinations (OSCE) were well researched, laborious and costly paper based method of exam delivery restricting international comparison. Cross-institutional comparison of OSCE Quality Assurance in Europe has never been done and due to wide spread electronic assessment analysis is now available.MethodsTwenty educational institutions across Europe using an electronic OSCE Management Information System where invited of which 8 confirmed to join a mutual comparison of Quality Assurance outcome. Two theories evaluate the quality of the observed test scores, the Classical psychometric theory (Cronbach’s alpha) and the Generalizability theory. Outcomes for both were compared for all universities including the Standard Error of Measurement (SEM) as well as cut-scores, Pass/Fail score and Global Rating Scores, Cronbach’s Alpha and related SEM (68% and 95% CI) and G-theory Coefficients with related absolute and relative SEM (68% and 95% CI). ResultsOutcomes differ between participating universities and observed marks contradict global rating of fail, borderline and excellent performance. G-theory coefficients and Standard Error of Measurement were lower and smaller compared to the classical approach using Cronbach’s Alpha as measure of reliability. The Classical psychometric based SEM varies from 2.8% to 11.2% respectively whereas the 95% CI equivalent varied from 9.2% up to 22% (on a 0 - 100% scale). The relative SEM from G-theory analysis varied from 3.1% to 7.0% for criterion-referenced marks, and the absolute SEM for norm-referenced marks varied from 3.8% to 7.8% respectively. The 95% CI around the relative and absolute SEMs values varied from 7.3% to 15.3%. More students failed the examination if the 95% CI is applied to the observed scores.ConclusionTo protect society and to improve educational decision making, the Standard Error of Measurement and associated confidence intervals needs to be embedded in EU assessment strategies to rule out ‘false positive Pass decisions’.

show abstract

Section: Discussionmentioning

confidence: 99%

Cross-institutional OSCE Quality Assurance in Europe using a mutual assessment strategy; are we equipped for it?

Kropmans¹,

Søfteland²,

Wijnalda³

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Because this validity aspect was not the focus of our study, future validation studies should apply multiple standard-setting methods and compare the processes as well as the results in order to validate the choice for a specific standard-setting method (e.g., for assessment in English education Hsieh 2013; for university education Çetin and Gelbal 2013; for medical education Yousuf et al 2015).…”

Section: Discussionmentioning

confidence: 99%

The standard setting process: validating interpretations of stakeholders

Kampa

Wagner

Köller

2019

Large-scale Assess Educ

View full text Add to dashboard Cite

Background: Stakeholders' interpretations of the findings of large-scale educational assessments can influence important decisions. In the context of educational assessment, standard-setting remains an especially critical element, because it is complex and largely unstandardized. Instruments established by means of standard-setting procedures such as proficiency levels (PL) therefore appear to be arbitrary to some degree. Owing to the significance such results take on, when they are communicated to stakeholders or the public, a thorough validation of this process seems crucial. In our study, ministry stakeholders intended to use PL established in an assessment of science abilities to obtain information about students' strengths and weaknesses regarding science abilities in general and specifically about the extent to which they were prepared for future science studies. The aim of our study was to investigate the validity arguments regarding these two intended interpretations.Methods: Based on a university science test administered to 3641 upper secondary students (Grade 13), a panel of nine experts set four cut scores using two variations of the Angoff method, the Yes/No Angoff method (multiple choice items) and the extended Angoff method (complex multiple choice items). We carried out t-tests, repeated measures ANOVA, G-studies and regression analyses to support the procedural, internal, external, and consequential validity elements regarding the aforementioned interpretations of the cut scores.Results: Our t-tests and G-studies showed that the intended use of the cut scores was valid regarding procedural and internal aspects of validity. These findings were called into question by the experts' lack of confidence in the established cut scores. Regression analyses including number of lessons taught and intended and pursued sciencerelated studies showed good external and poor consequential validity. Conclusion:The cut scores can be used as an indicator of 13th graders' strengths and weaknesses in science. They should not be used as an indicator for preparedness for science university studies. Since assessment formats are continually evolving and consequently leading to more complex designs, further research needs to be conducted on the application of new standard-setting methods to meet the challenges arising from this development. which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

show abstract

“…Since being introduced, the OSCE has become established as one of the leading assessment tools in medical schools and across many health professions education programs [ 2 – 7 ]. Previous studies provide extensive evidence supporting the reliability and validity of the OSCE [ 3 , 8 – 14 ]. To improve reliability examiners are commonly provided with a predetermined checklist to use when marking an examinee’s performance using categories from Fail to Distinction and at times numeric marks are attached to these categories [ 15 ].…”

Section: Introductionmentioning

confidence: 90%

“…To improve reliability examiners are commonly provided with a predetermined checklist to use when marking an examinee’s performance using categories from Fail to Distinction and at times numeric marks are attached to these categories [ 15 ]. Among the possible categories is a Borderline grade which describes a level of performance that is neither clear pass nor clear fail [ 14 , 16 , 17 ].…”

Section: Introductionmentioning

confidence: 99%

Borderline grades in high stakes clinical examinations: resolving examiner uncertainty

et al. 2018

View full text Add to dashboard Cite

BackgroundObjective Structured Clinical Exams are used to increase reliability and validity, yet they only achieve a modest level of reliability. This low reliability is due in part to examiner variance which is greater than the variance of students. This variance often represents indecisiveness at the cut score with apparent confusion over terms such as “borderline pass”. It is amplified by a well reported failure to fail.MethodsA borderline grade (meaning performance is neither a clear pass nor a clear fail) was introduced in a high stakes undergraduate medical clinical skills exam to replace a borderline pass grade (which was historically resolved as 50%) in a 4 point scale (distinction, pass, borderline, fail). Each Borderline grade was then resolved into a Pass or Fail grade by a formula referencing the difficulty of the station and the performance in the same domain by the student in other stations. Raw pass or fail grades were unaltered. Mean scores and 95%CI were calculated per station and per domain for the unmodified and the modified scores/grades (results are presented on error bars). To estimate the defensibility of these modifications, similar analysis took place for the P and the F grades which resulted from the modification of the B grades.ResultsOf 14,634 observations 4.69% were Borderline. Application of the formula did not impact the mean scores in each domain but the failure rate for the exam increased from 0.7 to 4.1%. Examiners and students expressed satisfaction with the Borderline grade, resolution formula and outcomes. Mean scores (by stations and by domains respectively) of students whose B grades were modified to P were significantly higher than their counterparts whose B grades were modified to F.ConclusionsThis study provides a feasible and defensible resolution to situations where the examinee’s performance is neither a clear pass nor a clear fail, demonstrating the application of the resolution of borderline formula in a high stakes exam. It does not create a new performance standard but utilises real data to make judgements about these small number of candidates. This is perceived as a fair approach to Pass/Fail decisions.

show abstract

Standard Setting Methods for Pass/Fail Decisions on High-Stakes Objective Structured Clinical Examinations: A Validity Study

Cited by 26 publications

References 51 publications

Cross-institutional OSCE Quality Assurance in Europe using a mutual assessment strategy; are we equipped for it?

Cross-institutional OSCE Quality Assurance in Europe using a mutual assessment strategy; are we equipped for it?

The standard setting process: validating interpretations of stakeholders

Borderline grades in high stakes clinical examinations: resolving examiner uncertainty

Contact Info

Product

Resources

About