Generalisability: a key to unlock professional assessment

Crossley, James; Davies, Helena; Humphris, Gerry; Jolly, Brian

doi:10.1046/j.1365-2923.2002.01320.x

Cited by 155 publications

(133 citation statements)

References 5 publications

Supporting

Mentioning

131

Contrasting

Order By: Relevance

“…The most traditionally used psychometric analysis for OSCEs is generalisability theory (Crossley, Davies, Humphris, & Jolly, 2002). Generalisability analysis works best when examiners are fully "crossed" with students and stations (i.e., all examiners observe all students at all stations); however, the model can produce useful data when some partial crossing is available (Shavelson, Webb, & Rowley, 1989).…”

Section: Introductionmentioning

confidence: 99%

Hawks, Doves and Rasch decisions: Understanding the influence of different cycles of an OSCE on students’ scores using Many Facet Rasch Modeling

Yeates

Sebok‐Syer

2016

Medical Teacher

View full text Add to dashboard Cite

Notes on contributors:Peter Yeates is a lecturer in medical education and consultant in acute and respiratory medicine. His research focuses on assessor variability and assessor cognition within health professionals' education.Stefanie S. Sebok-Syer is a postdoctoral Fellow at the Centre for Education Research and Innovation specializing in measurement, assessment, and evaluation. Her main interests include exploring the rating behaviour of assessors, particularly in high-stakes assessment contexts.

show abstract

Section: Introductionmentioning

confidence: 99%

Hawks, Doves and Rasch decisions: Understanding the influence of different cycles of an OSCE on students’ scores using Many Facet Rasch Modeling

Yeates

Sebok‐Syer

2016

Medical Teacher

View full text Add to dashboard Cite

show abstract

“…Content specificity underpins this variability and accounts for more variation than the rest of the identified sources combined. This is a common finding in generalizability study analyses 27 and illustrates the concept that the perceptions (and subsequent ratings) of any one interviewer along with his/her specific questions greatly affect an assessment's reliability; therefore, multiple interviewers should be sought. Using 8 interviewers within 8 separate interview stations was more reliable than having all 8 interviewers as 1 panel/station or even having 2 interviewers to 4 interview stations.…”

Section: Discussionmentioning

confidence: 78%

“…[25][26][27][28][29] The specific software program used herein was G_Strings IV, version 6.1.1 (Hamilton, ON, CA). The results output from a generalizability study will provide the relative contributions of different variance sources to variance in the entire data set, with a generalizability coefficient reported for reliability of the entire process (0.0-1.0 scale; similar to Cronbach's alpha).…”

Section: Phase Twomentioning

confidence: 99%

Improving Reliability of a Residency Interview Process

Peeters

Serres

Gundrum

2013

American Journal of Pharmaceutical Education

View full text Add to dashboard Cite

Objective. To improve the reliability and discrimination of a pharmacy resident interview evaluation form, and thereby improve the reliability of the interview process. Methods. In phase 1 of the study, authors used a Many-Facet Rasch Measurement model to optimize an existing evaluation form for reliability and discrimination. In phase 2, interviewer pairs used the modified evaluation form within 4 separate interview stations. In phase 3, 8 interviewers individuallyevaluated each candidate in one-on-one interviews. Results. In phase 1, the evaluation form had a reliability of 0.98 with person separation of 6.56; reproducibly, the form separated applicants into 6 distinct groups. Using that form in phase 2 and 3, our largest variation source was candidates, while content specificity was the next largest variation source. The phase 2 g-coefficient was 0.787, while confirmatory phase 3 was 0.922. Process reliability improved with more stations despite fewer interviewers per station-impact of content specificity was greatly reduced with more interview stations. Conclusion. A more reliable, discriminating evaluation form was developed to evaluate candidates during resident interviews, and a process was designed that reduced the impact from content specificity.Keywords: psychometrics, interview, residency, reliability INTRODUCTIONPostgraduate year 1 (PGY1) pharmacy residencies are increasingly prevalent in the United States; however, the pool of resident applicants has surpassed the number of available positions, 1 and professional organizations anticipate continued growth.2,3 Establishing a fair and objective process for selecting residents seems essential, and reliability of the process is a key characteristic in ensuring fairness in these candidate assessments. 4 An essential element during a resident selection process is the interview. Lack of validity, objectivity, reliability, and structure of the interview process exists in medical residency, 5-8 medical school 9 and pharmacy school admissions. 10Selecting the best candidates for pharmacy residency training is a difficult task. The ideal interview and selection process for residency candidates would be one that was efficient, objective, and produced reliable feedback/ information that could be used to make informed decisions. The survey questions asked and criteria used to make decisions appear to be fairly consistent among residency programs.11,12 Psychometric developments over the past few decades may help in increasing reliability of tools used in the interview process. The objective structured clinical examination (OSCE) was first discussed in 1979 and its interview "offspring," the multiple miniinterview (MMI), was described in 2003. Both showed promise for improving interview reliability. The overall concept and purpose of the MMI is to reduce the impact of content specificity, a concern in assessments, on the interview as compared to a traditional individual interview. [13][14][15] In the MMI, candidates rotate through interview stations where different...

show abstract

“…28 Using G-theory, important findings from OSCE-based evaluations have revealed that a global rating scale for each station (instead of a detailed checklist) was at least as reliable as a checklist over the multiple OSCE stations. 29 Regarding validity evidence, they also demonstrated that, among medical students, residents and physicians, global rating scales were able to detect changes in growing clinical expertise that checklists could not capture.…”

Section: Generalizability Theorymentioning

confidence: 99%

Educational Testing and Validity of Conclusions in the Scholarship of Teaching and Learning

Peeters

Beltyukova

Martin

2013

American Journal of Pharmaceutical Education

View full text Add to dashboard Cite

Validity and its integral evidence of reliability are fundamentals for educational and psychological measurement, and standards of educational testing. Herein, we describe these standards of educational testing, along with their subtypes including internal consistency, inter-rater reliability, and inter-rater agreement. Next, related issues of measurement error and effect size are discussed. This article concludes with a call for future authors to improve reporting of psychometrics and practical significance with educational testing in the pharmacy education literature. By increasing the scientific rigor of educational research and reporting, the overall quality and meaningfulness of SoTL will be improved.

show abstract

Generalisability: a key to unlock professional assessment

Cited by 155 publications

References 5 publications

Hawks, Doves and Rasch decisions: Understanding the influence of different cycles of an OSCE on students’ scores using Many Facet Rasch Modeling

Hawks, Doves and Rasch decisions: Understanding the influence of different cycles of an OSCE on students’ scores using Many Facet Rasch Modeling

Improving Reliability of a Residency Interview Process

Educational Testing and Validity of Conclusions in the Scholarship of Teaching and Learning

Contact Info

Product

Resources

About