Assessing the replication landscape in experimental linguistics

Abstract

Replications are an integral part of cumulative experimental science. Yet many scientific disciplines do not replicate much because novel confirmatory findings are valued over direct replications. To provide a systematic assessment of the replication landscape in experimental linguistics, the present study estimated replication rates for over 50,000 articles across 98 journals. We used automatic string matching using the Web of Science combined with in-depth manual inspections of 274 papers. The median rate of mentioning the search string “replicat*” was as low as 1.7 percent. Subsequent manual analyses of articles containing the search string revealed that only 4 percent of these contained a direct replication, i.e., a study that aims to arrive at the same scientific conclusions as an initial study by using exactly the same methodology. Less than half of these direct replications were performed by independent researchers. Thus our data suggest that only 1 in 1250 experimental linguistic articles contains an independent direct replication. We conclude that, similar to neighboring disciplines, experimental linguistics replicates very little, a state of affairs that should be reflected upon.

Semantic interference across word classes during lexical selection in Dutch

“…On a more general note, we wish to emphasise the importance of performing conceptual replication studies in psycholinguistics, a practice where the field is currently lagging behind compared to other fields within cognitive science (Kobrock & Roettger, 2023).…”

Advancements of Phonetics in the 21st century: Quantitative data analysis

“…A recent paper by Kobrock and Roettger (2023) finds that the rate of direct replications among experimental studies in linguistics is extremely low, potentially around 1 in 1250. The phonetics journals examined in their study do not seem to deviate from the general pattern in linguistics, suggesting similar rates.…”

Reproducible research practices and transparency across linguistics

“…Of the 360 articles that reported empirical studies, 4 articles (1.1%) were replications, according to the definition used in Kobrock and Roettger (2023). 5 Following Marsden, Morgan-Short, et al ( 2018), observed replication studies were categorized post hoc by classifying them according to the number of changes made.…”

“…5 Following Kobrock and Roettger (2023), we first searched for the search string "replicat" and if there was a hit, we examined the title and the abstract of the paper, the text before and after occurrences of the search term "replicat", the paragraph before the Methods section as well as the first paragraph of the Discussion section. If the authors explicitly claimed that (one of) their research aim(s) was to replicate the result or methods of an initial study, this article was treated as a replication.…”

“…are also prevalent in our field (Casillas, 2021;Kirby & Sonderegger, 2018;Laurinavichyute et al, 2022;Roettger, 2019;Vasishth et al, 2018). Recent meta-scientific assessments have investigated some aspects of research practices in language research, including the prevalence of direct replications (Kobrock & Roettger, 2023;Marsden, Morgan-Short, et al, 2018) and analytical flexibility (Coretta et al, 2023). Assessments of second language research Plonsky et al, 2015) reported limited sharing of materials (4-17%) and data (15%), and bilingualism researchers suggest poor availability of data, analysis, and materials in their subfield (Bolibaugh et al, 2021).…”