Faculty development on item writing substantially improves item quality

Abstract

The quality of items written for in-house examinations in medical schools remains a cause of concern. Several faculty development programs are aimed at improving faculty's item writing skills. The purpose of this study was to evaluate the effectiveness of a faculty development program in item development. An objective method was developed and used to assess improvement in faculty's competence to develop high quality test items. This was a quasi experimental study with a pretest-midtest-posttest design. A convenience sample of 51 faculty members participated. Structured checklists were used to assess the quality of test items at each phase of the study. Group scores were analyzed using repeated measures analysis of variance. The results showed a significant increase in participants' mean scores on Multiple Choice Questions, Short Answer Questions and Objective Structured Clinical Examination checklists from pretest to posttest (p < .0005). The effect sizes were 1.38, 3.84 and 2.20 for Multiple Choice Questions, Short Answer Questions and Objective Structured Clinical Examination, respectively. This study emphasizes that items written by faculty without faculty development are generally lacking in quality. It also provides evidence of the value of faculty development in improving the quality of items generated by faculty.

Development of knowledge tests for multi‐disciplinary emergency training: a review and an example

“…[11][12][13][14][15]19,30 We concluded that the face and content validity and the construct validity of the MCQ test were acceptable, as the test was able to discriminate between groups of participants expected to perform differently due to variations in knowledge and clinical skills.…”

“…Substantial insight in test development literature from textbooks and original scientific work is a necessary pre-requisite when embarking on test development. [11][12][13][14][15]19,20,29,30 Developing valid and reliable items requires competences within both test development and test statistics combined with in-depth knowledge on the content of the test. The template presented for item writing and the examples of MCQ items may prove useful for others who would like to develop a test and can potentially enhance the quality of tests by improving validity, correlation, and reliability.…”

Evaluación de reactivos de opción múltiple en medicina. Evidencia de validez de un instrumento

“…Identificar estos errores en las características cualitativas (relacionadas con la calidad en su elaboración), permitirá someterlos a revisión antes de ser aplicados, mejorando la construcción del instrumento. En la literatura existen diversas propuestas de recomendaciones o directrices para la elaboración de reactivos de opción múltiple, las cuales son muchas veces versiones resumidas y adaptadas de la propuesta de Haladyna para las necesidades de alguna institución; en nuestra búsqueda encontramos únicamente algunas listas de cotejo en idioma inglés para la evaluación de reactivos de opción múltiple, pero sin estudios que aporten evidencia de validez para su uso 13,14 , por lo que es necesario tener un instrumento que cuente con esta evidencia, para poder realizar una evaluación del aprendizaje más objetiva.…”

Collaborative Examination Item Review Process in a Team-Taught, Self-Care Sequence

“…3,8,9 Education literature indicates that intense training in item writing can improve the quality of faculty members' examination items. 10 However, whether individuals may improve overall quality of examinations during a semester course has not been evaluated. In this study, the authors implemented a course-level examination improvement process through a peer-led item-review program.…”