Using module analysis for multiple choice responses: A new method applied to Force Concept Inventory data

Abstract

We describe Module Analysis for Multiple Choice Responses (MAMCR), a new methodology for carrying out network analysis on responses to multiple choice assessments. This method is used to identify modules of non-normative responses which can then be interpreted as an alternative to factor analysis. MAMCR allows us to identify conceptual modules that are present in student responses that are more specific than the broad categorization of questions that is possible with factor analysis and to incorporate non-normative responses. Thus, this method may prove to have greater utility in helping to modify instruction. In MAMCR the responses to a multiple choice assessment are first treated as a bipartite, student X response, network which is then projected into a response X response network. We then use data reduction and community detection techniques to identify modules of non-normative responses. To illustrate the utility of the method we have analyzed one cohort of postinstruction Force Concept Inventory (FCI) responses. From this analysis, we find nine modules which we then interpret. The first three modules include the following: Impetus Force, More Force Yields More Results, and Force as Competition or Undistinguished Velocity and Acceleration. This method has a variety of potential uses particularly to help classroom instructors in using multiple choice assessments as diagnostic instruments beyond the Force Concept Inventory.

Gender fairness within the Force Concept Inventory

“…For many items, male and female scores were significantly different. For items 6,12,14,21,22,23,24, and 27, male and female difficulty scores were significantly different with a small effect size. This set of items contains most of the items which will be identified as significantly unfair by DIF analysis.…”

“…The Force Concept Inventory (FCI) [1] has been studied using tools such as factor analysis [2,3], item response theory [4,5], and network analysis [6]. Though these investigations have probed the structure and validity of the test, they have primarily treated student data as a single undifferentiated sample and have not studied gender effects.…”

Multidimensional item response theory and the Force Concept Inventory

“…These have included purely exploratory or descriptive methods such as factor analysis [7][8][9], module analysis [10], cluster analysis [11,12], item response theory [13][14][15][16], and item response curves [17,18]. The structure of student reasoning on the FCI has also been investigated by methods such as model analysis that require the input of a partial model of the concepts measured by the FCI [19].…”

Networks as integrated in research methodologies in PER

“…One of these modules is shown in Fig. 3 C. This has been labeled an 'impetus cluster', because many of the items resemble what has been called an impetus understanding of Newtonian mechanics [7]. However, we also discover that many of the response items are only weakly connected to this cluster.…”

“…Outwards Outwards Outwards [8]), we constructed a network of incorrect response items [7]. Two items are connected if at least one student has indicated both items.…”