Measuring individual differences in statistical learning: Current pitfalls and possible solutions

Abstract

Most research in Statistical Learning (SL) has focused on mean success rate of participants in detecting statistical contingencies at a group level. In recent years, however, researchers show increased interest in individual abilities in SL, either to predict other cognitive capacities or as a tool for understanding the mechanism underlying SL. Most, if not all of this research enterprise employs SL tasks that were originally designed for group-level studies. We argue that from an individual difference perspective, such tasks are psychometrically weak and sometimes even flawed. In particular, existing SL tasks have three major shortcomings: (1) the number of trials in the test phase is often too small (or, there is extensive repetitions of the same targets throughout the test), (2) a large proportion of the sample performs at chance level so that most of the data points reflect noise, and (3) test items following familiarization are all of the same type and identical level of difficulty. These factors lead to high measurement error, inevitably resulting in low reliability and thereby doubtful validity. Here we present a novel method specifically designed for the measurement of individual differences in visual SL. The novel task we offer displays substantially superior psychometric properties. We report data regarding the reliability of the task, and discuss the importance of the implementation of such tasks in future research.

Individual Differences in Statistical Learning: Conceptual and Measurement Issues

“…If these features exert a strong influence on the mechanisms that are engaged during statistical learning, the subtle differences in the tasks might result in weak inter-language correlations, a possibility which is potentially consistent with prior research (e.g., [42,64]). Such weak correlations could be difficult to detect (a sample size of close to 800 would be necessary to detect a correlation of 0.1 with power of 0.8; [65]). …”

Towards a theory of individual differences in statistical learning

“…If not, the task cannot differentiate between good and bad learners, and cannot reliably predict other cognitive capacities. As we have recently argued [12], most SL tasks that have been used for group-level studies do not withstand psychometric scrutiny. This is owing to a number of shortcomings, such as insufficient number of test trials or the difficulty of the task, which result in a large part of the sample performing at chance, and the lack of variability in test item difficulty.…”

“…These repetitions effects interfere with learning, thereby blurring the methodological separation between intended learning during familiarization and unintended learning that occurs during the test phase. 8 It is impossible to know whether responses reflect information acquired during learning or of overriding information presented by the repeated test items (see [12] for discussion).…”

“…could enable more powerful studies ( [11][12][13]; see also [14]). Note that 'individual differences' in the context of SL can in principle refer to any quantitative or qualitative differences between individual learners (i.e.…”

Statistical learning abilities and their relation to language

“…In parallel, other studies aim to further improve SL measurement by adding a continuous (or online) assessment of SL, focusing on the familiarization phase as subjects actually extract the regularities from the input stream. Behavioral studies using online measures typically use a task examining RTs to the stimuli during the exposure phase, and assess the magnitude of speed up to predictable compared to unpredictable items (Batterink, ; Kuppuraj, Duta, Thompson, & Bishop, ; Siegelman, Bogaerts, & Frost, ). Such online measures present an alternative to the reliance on the commonly used offline measures of SL (such as 2AFC tests), which only tap into performance after learning has occurred, and thus may interfere with the representations actually learned during familiarization (Siegelman, Bogaerts, Christiansen, et al, 2017).…”