Determining a hierarchy of correctness through student transitions on the FMCE

Investigating student learning and understanding of conceptual physics is a primary research area within Physics Education Research (PER). Multiple quantitative methods have been employed to analyze commonly used mechanics conceptual inventories: the Force Concept Inventory (FCI) and the Force and Motion Conceptual Evaluation (FMCE). Recently, researchers have applied network analytic techniques to explore the structure of the incorrect responses to the FCI identifying communities of incorrect responses which could be mapped on to common misconceptions. In this study, the method used to analyze the FCI, Modified Module Analysis (MMA), was applied to a large sample of FMCE pretest and post-test responses (Npre = 3956, Npost = 3719). The communities of incorrect responses identified were consistent with the item groups described in previous works. As in the work with the FCI, the network was simplified by only retaining nodes selected by a substantial number of students. Retaining nodes selected by 20% of the students produced communities associated with only four misconceptions. The incorrect response communities identified for men and women were substantially different, as was the change in these communities from pretest to post-test. The 20% threshold was far more restrictive than the 4% threshold applied to the FCI in the prior work which generated similar structures. Retaining nodes selected by 5% or 10% of students generated a large number of complex communities. The communities identified at the 10% threshold were generally associated with common misconceptions producing a far richer set of incorrect communities than the FCI; this may indicate that the FMCE is a superior instrument for characterizing the breadth of student misconceptions about Newtonian mechanics. *

Section: A the Fmce Instrumentmentioning

confidence: 99%

Exploring the structure of misconceptions in the Force and Motion Conceptual Evaluation with modified module analysis

Wells

Henderson

Traxler

et al. 2020

“…This interpretation of the slope parameters assumes that students who select correct options will also tend to select distractors that are more related to the latent trait than students who select fewer correct options [17]. As a result, some partial credit within questions may be allocated based on a student's progression through alternative world views while on a trajectory towards a Newtonian world view (see the discussion of question 17 in Sec.…”

Section: A Two-parameter Logistics Nominal Response Modelmentioning

confidence: 99%

“…Specifically, PIRT is a model-driven analysis which generates parameters for correct and incorrect response options, and then assigns students a latent trait ability score based on their selection of correct and incorrect response options [30]. Using the process proposed by Louis, Ricci, and Smith [17], this study generates a partial credit grading model for the FCI using a two-parameter logistic nominal response model (2PLNRM) from PIRT [31]. The partial credit scoring model present in this study is developed from the 2PLNRM, employed for use on the FCI with a dataset containing 19 745 student responses.…”

Section: Introductionmentioning

confidence: 99%

“…Louis, Ricci, and Smith, in Ref. [17], used the 2PLNRM to propose an ordering for the relative "correctness" of the distractors within individual questions on the Force and Motion Conceptual Assessment [32]. Indicated in their study is an assumption that must be made when using the 2PLNRM in order to generate an ordering for incorrect response options:…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Generating a growth-oriented partial credit grading model for the Force Concept Inventory

Eaton

Johnson

Willoughby

2019

Traditionally, multiple choice assessments are graded in a dichotomous manner, where selecting the correct option for a question awards 1 point and the selection of an incorrect option awards 0 points. As a result of this grading scheme, all incorrect response options are treated as being equally incorrect regardless of potential differences in their relative correctness, intentional or otherwise. We propose a partial credit grading model for the Force Concept Inventory (FCI) that allots points to the incorrect responses. This was done using slope parameters from the two-parameter logistics nominal response model (2PLNRM), a model from polytomous item response theory (PIRT). The resulting scores from the partial credit model represent student growth towards a proper Newtonian mindset, as measured by the FCI. Observations indicate that this model accounts for student progression through prominent misconceptions (i.e., impetus) as their world views become more Newtonian. As expected, we find that student total scores increase as a result of the model, but the average overall gains on the assessment are essentially unchanged. The data used in this analysis were maintained and organized by PhysPort and included about 20 000 responses from first semester introductory physics courses at multiple universities. Ultimately, this partial credit model allows instructors to more accurately gauge the growth of their students over the course of instruction. Additionally, as a result of these partial credit scores, we are able to identify potentially malfunctioning questions on the FCI that may be sources of error in measuring student abilities.

“…This assumption is consistent with previous work that has used item response curves (IRCs) to examine and rank incorrect responses on both the FCI and the FMCE [15][16][17][18]. We expand on this prior work by using a nested-logit item response theory (IRT) model to simultaneously estimate students' overall understanding of Newtonian mechanics (the IRT latent trait, or person parameter) and determine how closely each response choice correlates with a high level of understanding using the estimated parameters of the model [19][20][21][22][23][24]. Based on this assumption we would claim, for example, that a student who only incorrectly answers one question is more likely to choose a response that's almost correct than a student who answers 20 questions incorrectly.…”

Section: Introductionmentioning

confidence: 99%

Quantitatively ranking incorrect responses to multiple-choice questions using item response theory

Smith

Louis

Ricci

et al. 2020

Self Cite

Research-based assessment instruments (RBAIs) are ubiquitous throughout both physics instruction and physics education research. The vast majority of analyses involving student responses to RBAI questions have focused on whether or not a student selects correct answers and using correctness to measure growth. This approach often undervalues the rich information that may be obtained by examining students' particular choices of incorrect answers. In the present study, we aim to reveal some of this valuable information by quantitatively determining the relative correctness of various incorrect responses. To accomplish this, we propose an assumption that allow us to define relative correctness: students who have a high understanding of Newtonian physics are likely to answer more questions correctly and also more likely to choose better incorrect responses, than students who have a low understanding. Analyses using item response theory align with this assumption, and Bock's nominal response model allows us to uniquely rank each incorrect response. We present results from over 7,000 students' responses to the Force and Motion Conceptual Evaluation.