Development and validation of a multiple‐choice questionnaire‐based theoretical test in direct ophthalmoscopy

Abstract

Purpose Direct ophthalmoscopy can reveal systemic, neurologic and ophthalmic conditions, but is poorly mastered among young physicians. A theoretical test is needed to measure effect of educational interventions. We developed and gathered validity evidence for a multiple‐choice questionnaire (MCQ)‐based theoretical test in direct ophthalmoscopy. Methods The MCQ was developed by interviewing experts. Then, validity evidence was evaluated using Messick's validity framework. Content was ensured by inviting the experts to contribute in a Delphi‐like process. Response process was ensured by piloting and by streamlining all instructions. Then, the test was taken by ophthalmologists and by medical students without experience in direct ophthalmoscopy. Results were used to evaluate internal structure (item quality analysis and internal consistency), relations to other variables (correlation of test scores to experience level) and consequences (establishment of pass–fail score and the consequences of its use). Results The first phase of the study yielded 100 MCQs. In second phase, we identified that 60 items fulfilled predefined relevance and item quality requirements. These items demonstrated very high internal consistency (Cronbach's alpha = 0.95), significantly discriminated medical students from specialists (p < 0.001, independent samples t‐test) and the established pass–fail score of 50 (83%) correct answers resulted in no false positives (students passing) and no false negatives (specialists failing). A Decision study identified that sampling 15 items suffice for certification. Conclusion We developed and validated an MCQ‐based theoretical test in direct ophthalmoscopy that enables an evidence‐based approach to measuring, evaluating and certifying the theoretical knowledge necessary for direct ophthalmoscopy.

Modern Educational Simulation-Based Tools Among Residents of Ophthalmology: A Narrative Review

“…While sample size was rather low ( n = 21), Messick’s framework for validation was used which enhances evidence-based frames of the study. Jørgensen et al [ 41 ] aimed to develop and gather validity evidence for a multiple-choice questionnaire (MCQ)-based theoretical test in direct ophthalmoscopy and used Messick’s Validity Framework for evaluation of validity. A MCQ with 60 items was established through expert interviews and elimination rounds.…”

Theoretical Frameworks in Medical Education: Using a Systematic Review of Ophthalmology Education Research to Create a Theory of Change Model

“…The most common study types were prospective cohort [21][22][23][24][25][26][27][28][29] and cross-sectional. [30][31][32][33][34][35][36] Studies were most commonly conducted in the United States, 27,[37][38][39][40][41][42] Denmark, 23,28,36,43 and the United Kingdom. 25,26,44,45 Only 7 studies 22,26,31,32,35,44,46 had sample sizes over 50 participants (range 52-311).…”

“…25,26,44,45 Only 7 studies 22,26,31,32,35,44,46 had sample sizes over 50 participants (range 52-311). Ten studies 22,29,34,36,38,44,45,[47][48][49] included attending ophthalmologists, 9 studies 21,27,30,33,37,[39][40][41][42] included residents, 6 studies 24,26,31,32,35,46 included medical students, and 4 studies 23,25,28,43 included a mixed selection of participants. Seven educational intervention studies were conducted in a hospital/ clinic, 21,22,29,32,37,39,40 6 in an in-person classroom, 2 4 , 3 0 , 3 1 , 3 5 , 4 1 , 4 6 9 in simulation centers, 2 3 , 2 5 , 2 8 , 3 3 , 3 6 , 4 0 , 4 2 , 4 4 , 4 5 1 in a virtual classroom, 26 and 1 a...…”

Development of and Gathering Validity Evidence for a Theoretical Test in Contrast-Enhanced Ultrasound

“…In our study, all test participants had a relatively high level of US experience (equivalent to EFSUMB level I or higher), and thus our lowest proficiency group could be classified as "CEUS-naı ¨ve US intermediates." Several similar studies of theoretical test development within medical education compare test performance of medical students and specialists, which are the two extremes of proficiency level (Jørgensen et al 2019;Pietersen et al 2019). Naturally, when comparing very high and very low proficiency groups, the likelihood of observing a significant difference in test scores is higher, and the probability of false negatives and positives when establishing a pass/fail score is lower.…”