Estimating reliability for response-time difference measures: Toward a standardized, model-based approach

Abstract

A slowdown or a speedup in response times across experimental conditions can be taken as evidence of online deployment of knowledge. However, response-time difference measures are rarely evaluated on their reliability, and there is no standard practice to estimate it. In this article, we used three open data sets to explore an approach to reliability that is based on mixed-effects modeling and to examine model criticism as an outlier treatment strategy. The results suggest that the model-based approach can be superior but show no clear advantage of model criticism. We followed up these results with a simulation study to identify the specific conditions in which the model-based approach has the most benefits. Researchers who cannot include a large number of items and have a moderate level of noise in their data may find this approach particularly useful. We concluded by calling for more awareness and research on the psychometric properties of measures in the field.

Scaffolding comprehension with reading while listening and the role of reading speed and text complexity

“…As such, there was inherent clustering between observations based on participants and text parts. To account for this dependency in the data as a result of repeated measures, I analyzed the data using mixed-effects modeling, which can simultaneously model fixed and random effects (e.g., Hui & Wu, 2023;Linck & Cunnings, 2015). The outcome variable was specified as the logarithmic transformed comprehension score.…”

Reflecting on the use of response times to index linguistic knowledge in SLA

“…The first step in this direction is to properly examine the psychometric properties of RT differences. For example, there has already been work on estimating the reliability of RT differences using a model-based approach (Hui & Wu, 2024). If RT differences can be reliable, the next step is to more closely scrutinize their validity by, for example, correlating priming measures with existing vocabulary measures or by using RT differences to predict language performance.…”

“…Therefore, RT measures have been useful indirect tests that are believed to tap into implicit knowledge (Rebuschat, 2013). In summary, RT measures have been popular in SLA because they likely index the kind of lexical and grammatical knowledge that is readily available for relatively effortless, efficient processing and therefore authentic language use (e.g., Hui & Wu, 2024;Suzuki, 2017). Although there are various RT measures used in SLA, we summarize two broadly defined categories: primary and secondary RT measures.…”

“…On the contrary, constant scrutiny of reliability and validity should be carried out to ensure that the empirical base for theorization is strong (e.g., Cohen & Macaro, 2013;Hamrick, 2022). With some discussion on the reliability of RT measures already put forward (e.g., Hui & Wu, 2024), we focus more on the validity of RT measures in this paper.…”