Challenging the reliability and validity of cognitive measures: The case of the numerical distance effect

Number Compasison and Number Line Estimation Rely on Different Mechanisms

“…However, we have strong arguments to believe that both tasks we used are highly reliable and valid. First, Gilmore, Attridge and Inglis (2011) found significant split-half reliability coefficients in non-symbolic comparison tasks, very similar to the one used in this study (see also Maloney, Risko, Preston, Ansari, & Fugelsang, 2010;Sasanguie, Defever, Van den Bussche, & Reynvoet, 2011). Second, the fact that the performance of children on the non-symbolic number line task is significantly related to their performance on a symbolic number line task (e.g., Sasanguie, De Smedt et al, 2012) and to their math achievement scores (e.g., Sasanguie, De Smedt et al, 2012;Sasanguie, Van den Bussche et al, 2012), demonstrates both the reliability and the validity of the non-symbolic number line task.…”

Mental chronometry and individual differences: Modeling reliabilities and correlations of reaction time means and effect sizes

“…5), the larger effect size could have been responsible for the greater reliability of repetition priming, negating its support for the uncoordinated nature of semantic processes. Similarly, Maloney et al (2010) found much lower reliability for numerical distance effects obtained with numbers presented in symbolic formats (e.g., "4") than with those presented nonsymbolically (e.g., four squares), and these reliability differences may also have been due at least partly to the fact that the distance effect was much smaller with symbolic than with nonsymbolic stimuli (i.e., approximately 50 vs. 500 ms). Thus, as was the case with RT correlations, IDRT is useful in elucidating the many factors that need to be considered when interpreting changes in RT reliabilities.…”

“…For example, Corballis (2002) examined the correlation between two RT-based effects thought to depend on the time needed for interhemispheric transmission (i.e., redundancy gain and the crossed-uncrossed difference) in order to find out whether the two effects were mediated by a common neural pathway. Likewise, Maloney, Risko, Preston, Ansari and Fugelsang (2010) examined correlations of numerical distance effects obtained with different number formats in order to see whether these different effects assessed the same underlying numerical representations and comparison mechanisms. Similarly, Stolz, Besner and Carr (2005; see also Waechter, Stolz, & Besner, 2010) examined the intercorrelations of semantic priming effects across different RT sessions in order to assess the extent to which individual variation in these effects reflect systematic differences in semantic associations between individuals, as opposed to merely statistical noise.…”

Indexing the approximate number system

“…Weber fraction on three variants of the nonsymbolic comparison task, finding reliability coefficients varying between r = .4 and .8; Maloney, Risko, Preston, Ansari & Fugelsang (2010) found that the immediate test-retest reliability of an NDE(accuracy) measure was in the same range, r ≈ .6. Remarkably, Libertus et al (2012) found that the three month testretest reliability of their measure of individuals' Weber fractions was not significantly different to zero.…”