Item Wording Effects in Psychological Measures: Do Early Literacy Skills Matter?

Abstract

While the inclusion of both positively and negatively worded items is a common practice in scales, using positively and negatively worded items together may threaten the validity of a scale. Several studies have been devoted to investigating the effects of item wording methods. The current study investigated item wording effects on the responses of 4028 Turkish fifth-grade students, who responded to the Students Confidence in Mathematics (SCM) and Students Confidence in Science (SCS) scales. The role of early literacy-related variables (i.e., early literacy activities undertaken before primary school, student performance on reading literacy tasks upon entering primary school, and duration of the children's pre-primary school attendance) on item wording effects was also examined. The investigations were conducted using confirmatory factor analysis and the correlated trait-correlated method minus one CFA-CTC(M-1) model, derived from the correlated traits-correlated methods framework. The results indicate that significant item wording effects existed in both scales. Moreover, a significant and positive effect was found in both scales relating to early literacy activities undertaken before school, but no effects were found relating to student performance on reading literacy tasks upon entering primary school or duration of the children's pre-primary school attendance. Overall, the study suggests that researchers and practitioners should consider potential effects when including both positively and negatively worded items in scales, especially scales designed for younger students.

The effects of reverse items on psychometric properties and respondents’ scale scores according to different item reversal strategies

“…More exactly, regardless of which reversal strategy is used, the reverse items caused an artificial factor that can also be called a method factor in addition to the trait factor. Consistent with this result, many studies in the literature revealed that reverse items in scales measuring a single unitary construct distort the factor structure of the scale by generating a separate dimension, which leads to an unintentionally multidimensional factor solution (Bulut & Bulut;DiStefano & Motl, 2006;Dunbar et al, 2000;Hazlett-Stevens et al, 2004;Herche & Engelland, 1996;Knight et al, 1998;Pilotte & Gable, 1990;Spector et al, 1997).…”

“…Indeed, Schmitt and Allik (2005) and Hooper et al (2013) disclosed that reverse items were interpreted differently across countries. The mentioned contradiction may also be associated with the sample characteristics because the factors such as reading achievements (Bulut & Bulut, 2022;Michaelides, 2019), age levels (Bulut, 2021;Marsh 1986) and cognitive abilities (Gnambs & Schroeders, 2020) The results of the study indicated that internal consistency coefficient of Form-RN was lower than that of the other two versions. The facts that the reliability coefficient of Form-RN is lower than that of Form-P are coherent with those found by other researchers (Bourque & Shen, 2005;Carlson, 2011;Coleman, 2013;Ebesutani et al, 2012;Johnson et al, 2004;Roszkowski & Soven, 2010;Salazar, 2015;Suárez-Alvarez, 2018).…”

“…A review of the literature disclosed that research about reverse items has a long history, and most studies focused on the first dilemma mentioned previously. The effects of reverse items on validity, internal consistency, factorial structure and factor loadings, item correlations, missing values, and respondents' ability estimation (mean scores on the scale) are principal subjects emphasized in these studies (e.g., Ahlawat, 1985;Bergstrom & Lunz, 1998;Boley et al, 2021;Bolt et al, 2020;Bulut & Bulut, 2022;Bulut, 2021;Chamberlain & Cummings, 1984;Conrad et al, 2004;Dooden, 2014;Dueber et al, 2021;Guyatt et al, 1999;Herche & England, 1996;Hooper et al, 2013, İlhan & Güler, 2017Kula Kartal, 2021;Kula Kartal et al, 2022;Locker et al, 2013;Marsh, 1996;Schriesheim & Hill, 1981;Salazar, 2015;Schotte et al, 1996;Suárez-Alvarez et al, 2018;Vigil-Colet et al, 2020). Conversely, the number of studies seeking answer to the second dilemma regarding reverse items is more limited.…”