Best Practices Related to Examination Item Construction and Post-hoc Review

Abstract

Objective. To provide a practical guide to examination item writing, item statistics, and score adjustment for use by pharmacy and other health professions educators. Findings. Each examination item type possesses advantages and disadvantages. Whereas selected response items allow for efficient assessment of student recall and understanding of content, constructed response items appear better suited for assessment of higher levels of Bloom's taxonomy. Although clear criteria have not been established, accepted ranges for item statistics and examination reliability have been identified. Existing literature provides guidance on when instructors should consider revising or removing items from future examinations based on item statistics and review, but limited information is available on performing score adjustments. Summary. Instructors should select item types that align with the intended learning objectives to be measured on the examination. Ideally, an examination will consist of multiple item types to capitalize on the advantages and limit the effects of any disadvantages associated with a specific item format. Score adjustments should be performed judiciously and by considering all available item information. Colleges and schools should consider developing item writing and score adjustment guidelines to promote consistency.

Evaluation of validity and reliability of multiple-choice questions in second MBBS competency-based medical education-based pharmacology examination of medical institute of India

“…For most of the course-based exams, Cronbach's alpha value ranges from 0.6 to 0.8, and in general, the value should be at least 0.5. 10 In our study, the reliability of the test was 0.77 which means the given MCQs test is reliable.…”

“…A minimum value of d should be 0.15 for course-based examination while for standardized examination d value should be at least 0.3. 10 In the present study, there are 19 items (47.5 %) had moderate difficulty index and 4 items had higher difficulty index. In our study, 30 items have a discrimination index >0.15 having marginal to excellent discrimination index while 10 items have a poor discrimination index.…”

The Health Improvement Profile for people with severe mental illness: Feasibility of a secondary analysis to make international comparisons

“…Internal consistency of the HIP questionnaire for the four-country sample was calculated by the Kudar Richardson-20 procedure for binary variables using SPSS (Rudolph et al, 2019, SPSS, 2020.…”

“…The internal consistency (reliability) score for data (all four countries) was 0.751 for the full scale (18 HIP items) and ranged from 0.716 to 0.763 for individual items. This indicates acceptable internal consistency, as 0.6 to 0.8 are desired for KR-20 (Rudolph et al, 2019; Statistical Program for Social Sciences (SPSS), 2019). We note that the KR-20 does not assure criterion homogeneity, only internal consistency in this instance of administration of the HIP in the different samples (SPSS, 2019).…”

Getting Rigor Right

“…At the individual level, faculty members can engage in self-evaluation and personal reflection to continually refine the levels of academic rigor in their courses. Together with those in charge of programmed faculty development, faculty members can seek to improve their knowledge and skills in regard to establishing appropriate educational expectations and eliminating instances of contrived rigor, [12][13][14] writing fair test questions and evaluating these questions statistically, 15 evaluating student prior knowledge, 16 and using instructional methods that benefit underprepared learners. [17][18][19] Given the adverse consequences that can arise from either abandonment or accretion of academic rigor, it is important for individual faculty members and their programs to make an effort to identify and minimize gaps in knowledge and skills, maladaptive motivations, and any perverse incentives that may be working against maintaining appropriate levels of academic rigor in their pharmacy classrooms.…”