Towards a credibility revolution in bilingualism research: Open data and materials as stepping stones to more reproducible and replicable research

Abstract

The extent to which findings in bilingualism research are contingent on specific analytic choices, experimental designs, or operationalisations, is currently unknown. Poor availability of data, analysis code, and materials has hindered the development of cumulative lines of research. In this review, we survey current practices and advocate a credibility revolution in bilingualism research through the adoption of minimum standards of transparency. Full disclosure of data and code is necessary not only to assess the reproducibility of original findings, but also to test the robustness of these findings to different analytic specifications. Similarly, full provision of experimental materials and protocols underpins assessment of both the replicability of original findings, as well as their generalisability to different contexts and samples. We illustrate the review with examples where good practice has advanced the agenda in bilingualism research and highlight resources to help researchers get started.

Promoting computationally reproducible research in applied linguistics: Recommended practices and considerations

“…Sharing supplementary information is important in open science practices as it offers many benefits. Bolibaugh et al (2021) discussed three advantages of sharing data and analysis code and one advantage of sharing study materials and protocols. First, sharing data and code allows researchers to verify study findings.…”

“…Second, Bolibaugh et al (2021) argued that sharing data and code makes it possible to examine the analytic robustness of results. When collecting and analyzing data, researchers must decide whether to collect more data, how to treat missing data and outliers, whether to collapse scales with a small number of responses, among other things.…”

“…Despite the importance of reproducible research, only recently has it been gaining traction as a topic worthy of consideration and empirical attention (see Bolibaugh et al, 2021;Gass et al, 2021). This comes from findings that research is not as reproducible as researchers think.…”

“…Toward that end, building on and extending recommended open science research practices (e.g., Bolibaugh et al, 2021;Marsden, 2020;Plonsky, in press), we discuss three issues: (1) where and how to share supplementary information needed to reproduce study results, (2) what information to present and how to do so in a reproducible manner, and (3) how to create simulated data if the original data cannot be shared due to proprietary or ethical reasons. The first two are closely related to the importance of sharing supplementary information as will be reviewed below (Bolibaugh et al, 2021). We selected the third issue to address situations where data cannot be shared despite the authors' wish to do so.…”

From ‘No, she does’ to ‘Yes, she does’: Negation processing in negative yes–no questions by Chinese users of English

“…The full dataset with responses for each participant in each condition, as well as the materials used, are made available to the research community on the project website https://osf.io/x4536/. These steps were taken to promote reproducible and replicable practices in bilingualism research more generally (Bolibaugh, Vanek, & Marsden, 2021), following examples from language-modulated event cognition research with Mandarin-English bilinguals in particular (Vanek, 2020;Tang, Vanek & Roberts, 2020). Then, we proceeded with a forward variable selection and zoomed in on the negative questions, comparing a model including Group with a reduced model without Group in the data for negative questions only.…”