Guidance for high-stakes testing within pharmacy educational assessment

Peeters, Michael J.; Cor, M. Ken

doi:10.1016/j.cptl.2019.10.001

Cited by 14 publications

(29 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Moreover, we used a conventional high-stakes reliability cutoff of 0.8. 1,3 Using Classical Test Theory for Individual Exams. Using the KR-20 for internal consistency, a reliability coefficient for students' scores was reported for each individual exam by ExamSoft (ExamSoft Worldwide, Dallas TX).…”

Section: Approaches To Reliabilitymentioning

confidence: 99%

Validation Evidence from using Generalizability Theory in a Basic-Science Course

et al. 2021

View full text Add to dashboard Cite

Description of the Problem: Reliability is critical validation evidence on which to base high-stakes decision-making. Many times, one exam in a didactic course may not be acceptably reliable on its own. But how much might multiple exams add when combined together? The Innovation: To improve validation evidence towards high-stakes decision-making, Generalizability Theory (G-Theory) can combine reliabilities from multiple exams into one composite-reliability (G_String IV software). Further, G-Theory decision-studies can illustrate changes in course-grade reliability, depending on the number of exams and exam-items. Critical Analysis: 101 first-year PharmD students took two midterm-exams and one final-exam in a pharmaceutics course. Individually, Exam1 had 50MCQ (KR-20=0.69), Exam2 had 43MCQ (KR-20=0.65), and Exam3 had 67MCQ (KR-20=0.67). After combining exam occasions using G-Theory, the composite-reliability was 0.71 for overall course-grades—better than any exam alone. Remarkably, increased numbers of exam occasions showed fewer items per exam were needed, and fewer items over all exams, to obtain an acceptable composite-reliability. Acceptable reliability could be achieved with different combinations of number of MCQs on each exam and number of exam occasions. Implications: G-Theory provided reliability critical validation evidence towards high-stakes decision-making. Final course-grades appeared quite reliable after combining multiple course exams—though this reliability could and should be improved. Notably, more exam occasions allowed fewer items per exam and fewer items over all the exams. Thus, one added benefit of more exam occasions for educators is developing fewer items per exam and fewer items over all exams.

show abstract

Section: Approaches To Reliabilitymentioning

confidence: 99%

Validation Evidence from using Generalizability Theory in a Basic-Science Course

et al. 2021

View full text Add to dashboard Cite

show abstract

“…This is especially so for high-stakes decision-making that involves use of test scores from a learning assessment (e.g., NAPLEX score for pharmacist licensing). 12 Corresponding author: Michael J. Peeters, PharmD, PhD, FCCP, BCPS; University of Toledo College of Pharmacy & Pharmaceutical Sciences Email: michael.peeters@utoledo.edu…”

Section: Why Is This Innovation Important?mentioning

confidence: 99%

Moving beyond Cronbach’s Alpha and Inter-Rater Reliability: A Primer on Generalizability Theory for Pharmacy Education

Peeters

2021

Innov Pharm

View full text Add to dashboard Cite

Background: When available, empirical evidence should help guide decision-making. Following each administration of a learning assessment, data becomes available for analysis. For learning assessments, Kane’s Framework for Validation can helpfully categorize evidence by inference (i.e., scoring, generalization, extrapolation, implications). Especially for test-scores used within a high-stakes setting, generalization evidence is critical. While reporting Cronbach’s alpha, inter-rater reliability, and other reliability coefficients for a single measurement error are somewhat common in pharmacy education, dealing with multiple concurrent sources of measurement error within complex learning assessments is not. Performance-based assessments (e.g., OSCEs) that use raters, are inherently complex learning assessments. Primer: Generalizability Theory (G-Theory) can account for multiple sources of measurement error. G-Theory is a powerful tool that can provide a composite reliability (i.e., generalization evidence) for more complex learning assessments, including performance-based assessments. It can also help educators explore ways to make a learning assessment more rigorous if needed, as well as suggest ways to better allocate resources (e.g., staffing, space, fiscal). A brief review of G-Theory is discussed herein focused on pharmacy education. Moving Forward: G-Theory has been common and useful in medical education, though has been used rarely in pharmacy education. Given the similarities in assessment methods among health-professions, G-Theory should prove helpful in pharmacy education as well. Within this Journal and accompanying this Idea Paper, there are multiple reports that demonstrate use of G-Theory in pharmacy education.

show abstract

“…With delaying a student's progression through their PharmD program seen as one potential high-stakes decision-making scenario, validation evidence by way of reliability can be foundational. 1,8 Practically speaking, recall that the commonly-accepted threshold for high-stakes testing is 0.8; 1 a reliability below this should be supplemented with other learning assessment data.…”

Section: Key Issuesmentioning

confidence: 99%

Providing Validation Evidence for a Clinical-Science Module: Improving Testing Reliability with Quizzes

2021

View full text Add to dashboard Cite

Description of the Problem: High-stakes decision-making should have sound validation evidence; reliability is vital towards this. A short exam may not be very reliable on its own within didactic courses, and so supplementing it with quizzes might help. But how much? This study’s objective was to understand how much reliability (for the overall module-grades) could be gained by adding quiz data to traditional exam data in a clinical-science module. The Innovation: In didactic coursework, quizzes are a common instructional strategy. However, individual contexts/instructors can vary quiz use formatively and/or summatively. Second-year PharmD students took a clinical-science course, wherein a 5-week module focused on cardiovascular therapeutics. Generalizability Theory (G-Theory) combined seven quizzes leading to an exam into one module-level reliability, based on a model where students were crossed with items nested in eight fixed testing occasions (mGENOVA used). Furthermore, G-Theory decision-studies were planned to illustrate changes in module-grade reliability, where the number of quiz-items and relative-weighting of quizzes were altered. Critical Analysis: One-hundred students took seven quizzes and one exam. Individually, the exam had 32 multiple-choice questions (MCQ) (KR-20 reliability=0.67), while quizzes had a total of 50MCQ (5-9MCQ each) with most individual quiz KR-20s less than or equal to 0.54. After combining the quizzes and exam using G-Theory, estimated reliability of module-grades was 0.73; improved from the exam alone. Doubling the quiz-weight, from the syllabus’ 18% quizzes and 82% exam, increased the composite-reliability of module-grades to 0.77. Reliability of 0.80 was achieved with equal-weight for quizzes and exam. Next Steps: Expectedly, more items lent to higher reliability. However, using quizzes predominantly formatively had little impact on reliability, while using quizzes more summatively (i.e., increasing their relative-weight in module-grade) improved reliability further. Thus, depending on use, quizzes can add to a course’s rigor.

show abstract

Guidance for high-stakes testing within pharmacy educational assessment

Cited by 14 publications

References 6 publications

Validation Evidence from using Generalizability Theory in a Basic-Science Course

Validation Evidence from using Generalizability Theory in a Basic-Science Course

Moving beyond Cronbach’s Alpha and Inter-Rater Reliability: A Primer on Generalizability Theory for Pharmacy Education

Providing Validation Evidence for a Clinical-Science Module: Improving Testing Reliability with Quizzes

Contact Info

Product

Resources

About