Why normalized gain should continue to be used in analyzing preinstruction and postinstruction scores on concept inventories

Abstract

Recently, Nissen et al. argued in this journal for the use of Cohen's d, in place of the more commonly used normalized gain, in the analysis of preinstruction and postinstruction scores on concept inventories used to measure the effectiveness of instruction. Their reason for advocating such a change is that they say normalized gains are "prescore biased." We provide five examples, including one cited by Nissen, that show no prescore bias when data are carefully analyzed, demonstrating that the problem with their analysis is omitted variable bias. We show that Cohen's d is less informative than normalized gain when used as a single parameter measure of teaching effectiveness, even though, as Nissen points out, d is more widely used in other fields. We believe that physics education researchers should continue to use normalized gain to assess educational effectiveness of pedagogy. However, because different student populations can have significantly different responses to the same pedagogy, in any interpretation of normalized gain, it is important to consider a measure of the abilities of the students. In analyzing normalized gains for the Force Concept Inventory (FCI), average scores on either Lawson's Test of Scientific Reasoning Ability or the SAT should be considered, because these scores are strongly correlated with normalized gain, indicating student abilities may have a greater impact on the gains achieved in a class than the specific pedagogy used.

Student-Generated Infographics for Learning Green Chemistry and Developing Professional Skills

“…To assess the learning gains of the students, an average normalized learning gain of 0.36 was calculated using Hake’s method (eq ). , Using Hake’s demarcation, this would correspond to a medium learning gain as a cognitive outcome of this activity. , …”

The Effectiveness of Web-Based Recitation Program on Improving Students’ Conceptual Understanding in Fluid Mechanics

“…The effectiveness of the program is analyzed based on differences in the N-gain of pretest-posttest score (Coletta & Steinert, 2020) between the experimental and the control groups, and the response of the experimental class students to the program. Pretest was administered after the three groups of students have learned fluid mechanics through regular lecture sessions, and the posttest was administered after the experimental groups have finished learning with the help of the program.…”

Modelling instruction effect with different reasoning ability on physics conceptual understanding by controlling the prior knowledge

“…One of the factors is the scientific reasoning skill possessed by the students even before they came to school, and the second is how their perception regarding Physics itself (Ding, 2014). Also, this statement is on the same path with early research which was done by Coletta that has confirmed about the causal effect between the scientific reasoning ability and the students' advantages in Newton Law concept (Coletta et al, 2008(Coletta et al, , 2012Coletta & Phillips, 2005;Coletta & Steinert, 2020).…”

“…The importance of Modelling in supporting the learning process is inevitably will elevate students' understanding concept ability. In the comparative analyses of inventory concept test score (FCI), it is shown that the students whom are taught by teachers who use Modelling method, have higher average score and higher FCI post test score comparing to the students who only receive traditional treatment during the physics learning process (Coletta et al, 2012;Coletta & Steinert, 2020;Jackson et al, 2008).…”