A methodological synthesis of self-paced reading in second language research

Abstract

Self-paced reading tests (SPRs) are being increasingly adopted by second language (L2) researchers. Using SPR with L2 populations presents specific challenges, and its use is still evolving in L2 research (as well as in first language research, in many respects). Although the topic of several narrative overviews (Keating & Jegerski, 2015; Roberts, 2016), we do not have a comprehensive picture of its usage in L2 research. Building on the growing body of systematic reviews of research practices in applied linguistics (e.g., Liu & Brown, 2015; Plonsky, 2013), we report a methodological synthesis of the rationales, study contexts, and methodological decision making in L2 SPR research. Our comprehensive search yielded 74 SPRs used in L2 research. Each instrument was coded along 121 parameters, including: reported rationales and study characteristics, indicating the scope and nature of L2 SPR research agendas; design and analysis features and reporting practices, determining instrument validity and reliability; and materials transparency, affecting reproducibility and systematicity of agendas. Our findings indicate an urgent need to standardize the use and reporting of this technique, requiring empirical investigation to inform methodological decision making. We also identify several areas (e.g., study design, sample demographics, instrument construction, data analysis, and transparency) where SPR research could be improved to enrich our understanding of L2 processing, reading, and learning.

Multiple Regression in L2 Research: A Methodological Synthesis and Guide to Interpreting R2 Values

“…This scheme acted as the data collection instrument to help us understand how MRs were used and reported in L2 research articles. To the best of our knowledge, no previous coding scheme had been developed for examining practices related to MR. Consequently, we created the coding scheme for this study based on (a) the different checklists and recommendations in statistical guides (Field, ; Hair et al., ; Keith, ; Kelley & Maxwell, ; Larson–Hall, ; Meyers, Gamst, & Guarino, ; Pituch & Stevens, ; Plonsky, ; Plonsky & Oswald, ; Tabachnick & Fidell, ), (b) the recommendations in the publication manual of the American Psychological Association (APA, ), (c) suggestions, recommendations, and findings found in different recently published L2 research reviews (Larson–Hall & Plonsky, ; Plonsky, , , ), and (d) previous methodologically oriented syntheses (e.g., Lindstromberg, ; Marsden et al., ; Plonsky & Gonulal, ). As is typical in systematic reviews, the coding scheme underwent several rounds of development and piloting.…”

Reexamining the Role of Vision in Second Language Motivation: A Preregistered Conceptual Replication of You, Dörnyei, and Csizér (2016)

“…(; henceforth, the initial study) in order to contextualize the design of our conceptual replication as well as to provide a state‐of‐the‐art account of the use of SEM in language learning research. The initial study did not fully report several technical points—an observation made elsewhere about a great deal of L2 research (e.g., Al‐Hoorie & Vitta, in press; Larson‐Hall & Plonsky, ; Marsden, Morgan‐Short, Thompson, & Abugaber, ; Marsden, Thompson, & Plonsky, )—making it impossible for us to offer a complete evaluation of the validity of the results, which thus emphasized the need for this replication. Following Marsden, Morgan‐Short, Thompson, and Abugaber’s () recommendation to spell out deviations from a replicated study as fully as possible, we have surveyed these methodological issues from the initial study in order to point out potential departures in our methodology from that of the initial study.…”

A Meta-Analysis of Sensitivity to Grammatical Information During Self-Paced Reading: Towards a Framework of Reference for Reading Time Effect Sizes

“…In addition to investigating the substantive issues above, an important, more general purpose of the current study was to investigate the feasibility and usefulness of calculating and interpreting effect sizes in L2 reading time research. To date, null hypothesis significance testing (NHST; e.g., analyses of variance, t-tests and, more recently, regression models with mixed effects) represents the default option for analysing data in L2 SPR (Marsden, Thompson, & Plonsky, 2018) and L2 research generally (Plonsky, 2013(Plonsky, , 2015. The main concerns about NHST include: (1) because p is influenced by both the relationship and sample size being measured, it may lead to false conclusions.…”

“…Despite these potential benefits, the d family of effect sizes has rarely been reported in L2 reading time studies to date, so we do not have a clear picture of what constitutes a 'small' or 'large' effect. (For exceptions, see Brysbaert & Stevens, 2018;Jegerski, 2018;Marsden, Thompson, & Plonsky, 2018. ) Illustrating two consequences of this problem, we note that in Adesope, Lavin, Thompson, & Ungerleider's (2010) meta-analysis, data had to be extracted to calculate effect sizes for the two reading time studies they included and, potentially more concerning, these effect sizes were aggregated with effects from a different type of data, test scores (accuracy, judgements etc.).…”

“…The search for studies involved two phases. First, we extracted studies that met the criteria above from Marsden, Thompson and Plonsky's (2018) methodological synthesis of L2 SPR research, for which the search finished in June 2016. Our second phase involved both updating that body of studies to the following 12 months 5 and extending it, given that it had only included journal articles.…”