Assessing the Dimensionality and Factor Structure of Multiple-Choice Exams

Abstract

Examination of a test's internal structure can be used to identify what domains or dimensions are being measured, identify relationships between the dimensions, provide evidence for hypothesized multidimensionality and test score interpretations, and identify construct-irrelevant variance. The purpose of this research is to provide a methodological comparison of three methods for assessing the dimensionality and factor structure for binary scored exams in which guessing may be operating. The factor analytic methods in Mplus, NOHARM, and TESTFACT are compared in the context of analyzing the Multistate Bar Examination, a multiple-choice exam reflecting a number of complex integrated content domains. Results indicate that the methods support similar factor solutions. However, support for solutions with more factors and more indicators per factor is only obtained when guessing is modeled. The implications for evaluating the internal structure of multiple-choice tests are discussed.

A Graphical Approach to Evaluating Equating Using Test Characteristic Curves

“…Factor analyses applied to these data suggested that there was a single prominent dimension measured by the MBE. The finding of a single prominent dimension for the MBE is consistent with Stone and Yeh (2006).…”

Post-examination interpretation of objective test data: Monitoring and improving the quality of high-stakes examinations: AMEE Guide No. 66

“…For example, if there were no missing question responses identified, this would suggest that students either had good knowledge or were guessing for some questions. Conversely, if there were missing question responses, this might be either an indication of an inadequate time for completing the examination, a particularly hard exam or negative marking is being used (Stone & Yeh 2006;Reeve et al 2007).…”