Coming of age: the past, present, and future of quantitative SLA research

Gass, Susan M.; Loewen, Shawn; Plonsky, Luke

doi:10.1017/s0261444819000430

Cited by 85 publications

(60 citation statements)

References 57 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…I also demonstrated these solutions using actual L2 meta-analysis data. Realizing this effort under the methodological reform movement that is currently taking place in L2 research (Gass et al, 2020;Norouzian & Plonsky, 2018a, 2018bNorouzian et al, 2018Norouzian et al, , 2019, I hope that L2 meta-analysts routinely employ the practical methods detailed in the present article. It is also my hope that L2 journals require meta-analytic research submitted for publication to provide the key IRR information needed for replicability purposes.…”

Section: Discussionmentioning

confidence: 99%

Interrater Reliability in Second Language Meta-Analyses

Norouzian

2021

Stud Second Lang Acquis

View full text Add to dashboard Cite

There has recently been a surge of interest in improving the replicability of second language (L2) research. However, less attention is paid to replicability in the context of L2 meta-analyses. I argue that conducting interrater reliability (IRR) analyses is a key step toward improving the replicability of L2 meta-analyses. To that end, I first discuss the foundation of IRR in the context of meta-analytic research. Second, I introduce two IRR measures, S index and Specific Agreement, which aid in improving the replicability of meta-analytic research. Third, I offer a flexible R program, meta_rate, to facilitate the conduct of IRR analyses for L2 meta-analyses. Fourth, I apply the R program to an actual meta-analytic L2 coding sheet to demonstrate the practical use of the IRR methods discussed. Finally, I provide interpretive guidelines to assist both L2 meta-analysts and journals with the transparent reporting of the IRR findings.

show abstract

Section: Discussionmentioning

confidence: 99%

Interrater Reliability in Second Language Meta-Analyses

Norouzian

2021

Stud Second Lang Acquis

View full text Add to dashboard Cite

show abstract

“…Instead of ignoring this state of affairs, we acknowledge that this tension has practical implications for researchers, and we will consider several options for maintaining statistical rigor as well as ecological validity. However, we argue that the common current practice of treating small samples as unproblematic, and simply listing sample size as one limitation to address in future studies, is no longer sufficient in light of increased attention in ISLA to general, as well as quantitative, methodological rigor (e.g., Byrnes, 2013;Gass, Loewen, & Plonsky, 2020;Marsden & Plonsky, 2018).…”

Section: What Is a Small Sample?mentioning

confidence: 94%

Small Samples in Instructed Second Language Acquisition Research

Loewen

Hui

2021

The Modern Language Journal

Self Cite

View full text Add to dashboard Cite

This commentary discusses the issue of small samples in instructed second language acquisition research. We discuss the current state of affairs, and consider the disadvantages of small samples. We also explore other considerations regarding sample size, such as research ethics and ecological validity. We present a range of recommendations for researchers to address the issue of small samples. Recommendations include (a) engaging in sample size planning, (b) opting for appropriate designs and analytical approaches, (c) reporting the constraints that sample size places on the interpretation of findings, and (d) cultivating a culture of collaboration and open science.

show abstract

“…At the same time, it is somewhat disheartening that such willingness seems to be more the exception than the rule. As applied linguistics continues to embrace open science practices such as open materials and study preregistration (see Gass, Loewen, & Plonsky, in press; Marsden, Morgan-Short, Trofimovich, & Ellis, 2018; Marsden & Plonsky, 2018), we hope that more researchers take steps toward transparency throughout their workflow. Doing so will not only allow for more robust findings in studies such as ours, but it will also allow for greater interpretability of and trust in empirical findings.…”

Section: Future Researchmentioning

confidence: 99%

Outliers in L2 Research in Applied Linguistics: A Synthesis and Data Re-Analysis

Nicklin

Plonsky

2020

Ann Rev Appl Linguist

Self Cite

View full text Add to dashboard Cite

Data from self-paced reading (SPR) tasks are routinely checked for statistical outliers (Marsden, Thompson, & Plonsky, 2018). Such data points can be handled in a variety of ways (e.g., trimming, data transformation), each of which may influence study results in a different manner. This two-phase study sought, first, to systematically review outlier handling techniques found in studies that involve SPR and, second, to re-analyze raw data from SPR tasks to understand the impact of those techniques. Toward these ends, in Phase I, a sample of 104 studies that employed SPR tasks was collected and coded for different outlier treatments. As found in Marsden et al. (2018), wide variability was observed across the sample in terms of selection of time and standard deviation (SD)-based boundaries for determining what constitutes a legitimate reading time (RT). In Phase II, the raw data from the SPR studies in Phase I were requested from the authors. Nineteen usable datasets were obtained and re-analyzed using data transformations, SD boundaries, trimming, and winsorizing, in order to test their relative effectiveness for normalizing SPR reaction time data. The results suggested that, in the vast majority of cases, logarithmic transformation circumvented the need for SD boundaries, which blindly eliminate or alter potentially legitimate data. The results also indicated that choice of SD boundary had little influence on the data and revealed no meaningful difference between trimming and winsorizing, implying that blindly removing data from SPR analyses might be unnecessary. Suggestions are provided for future research involving SPR data and the handling of outliers in second language (L2) research more generally.

show abstract

Coming of age: the past, present, and future of quantitative SLA research

Cited by 85 publications

References 57 publications

Interrater Reliability in Second Language Meta-Analyses

Interrater Reliability in Second Language Meta-Analyses

Small Samples in Instructed Second Language Acquisition Research

Outliers in L2 Research in Applied Linguistics: A Synthesis and Data Re-Analysis

Contact Info

Product

Resources

About