Replicability and reproducibility of research results for human language technology: introducing an LRE special section

Branco, António; Cohen, K. Bretonnel; Vossen, Piek; Ide, Nancy; Calzolari, Nicoletta

doi:10.1007/s10579-017-9380-0

Cited by 24 publications

(18 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the field of L2 research, given the importance of replication and the 50 calls for replication in L2 research that we identified, one might expect a substantial number of published replication studies by now. However, a perceived lack of prestige, excitement, and originality of replication plagues L2 research (Porte, ), as it does other disciplines (Berez‐Kroeker et al., ; Branco, Cohen, Vossen, Ide, & Calzolari, ; Chambers, ; Schmidt, ), and these perceptions are thought to have caused, at least in part (directly or indirectly), alleged low rates and a poor quality of published replication studies. However, a systematic metascience on replication research has not yet been established in the field of L2 research, leaving a poor understanding of the actual number and nature of replication studies that have been published.…”

Section: Introductionmentioning

confidence: 99%

Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field

et al. 2018

View full text Add to dashboard Cite

Despite its critical role for the development of the field, little is known about replication in second language (L2) research. To better understand replication practice, we first provide a narrative review of challenges related to replication, drawing on recent developments in psychology. This discussion frames and motivates a systematic review, building on syntheses of replication in psychology, education, and L2 research. We coded 67 self-labeled L2 replication studies found across 26 journals for 136 characteristics. We estimated a mean rate of 1 published replication study for every 400 articles, with a mean of 6.64 years between initial and replication studies and a mean of 117 citations of the initial study before a replication was published. Replication studies had an annual mean of 7.3 citations, much higher than averages in linguistics and education. Overlap in authorship between initial and replication studies and the availability of the initial materials both increased the likelihood of a replication supporting the initial findings. Our sample contained no direct (exact) replication attempts, and changes made to initial studies were numerous and wide ranging, which likely obscured, if not undermined, the interpretability of replication studies. To improve the amount and quality of L2 replication research, we propose 16 recommendations relating to rationale, nomenclature, design, infrastructure, and incentivization for collaboration and publication.

show abstract

Section: Introductionmentioning

confidence: 99%

Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field

et al. 2018

View full text Add to dashboard Cite

show abstract

“…On a surface level at least, it would seem that reproducibility has been part of the ML research agenda for quite a while. Journals have started special issues (Branco et al 2017), conference workshops (AAAI, NeurIPS, ICLR, ICML), and special tracks (e.g., COLING) to promote these efforts. We have seen heavy use of open access pre-prints (arXiv), code repositories (GitHub, Papers With Code), and e-publishing (Distill).…”

Section: Main Challenge: Inertia Of Cultural Changementioning

confidence: 99%

Towards Replication in Computational Cognitive Modeling: a Machine Learning Perspective

et al. 2019

View full text Add to dashboard Cite

The suggestions proposed by Lee et al. to improve cognitive modeling practices have significant parallels to the current best practices for improving reproducibility in the field of Machine Learning. In the current commentary on "robust modeling in cognitive science", we highlight the practices that overlap and discuss how similar proposals have produced novel ongoing challenges, including cultural change towards open science, the scalability and interpretability of required practices, and the downstream effects of having robust practices that are fully transparent. Through this, we hope to inform future practices in computational modeling work with a broader scope.

show abstract

“…On a surface level at least, it would seem that reproducibility has been part of the ML research agenda for quite a while. Journals have started special issues (Branco et al, 2017), conferences workshops (AAAI, NeurIPS, ICLR, ICML) and special tracks (e.g. COLING) to promote these efforts.…”

Section: Main Challenge: Inertia Of Cultural Changementioning

confidence: 99%

Towards Replication in Computational Cognitive Modeling: A Machine Learning Perspective

Emmery¹,

Kádár²,

Wiltshire³

et al. 2019

Preprint

View full text Add to dashboard Cite

The suggestions proposed by Lee et al. to improve cognitive modeling practices have significant parallels to the current best practices for improving reproducibility in the field of Machine Learning. In the current commentary on `Robust modeling in cognitive science', we highlight the practices that overlap and discuss how similar proposals have produced novel ongoing challenges, including cultural change towards open science, the scalability and interpretability of required practices, and the downstream effects of having robust practices that are fully transparent. Through this, we hope to inform future practices in computational modeling work with a broader scope.

show abstract

Replicability and reproducibility of research results for human language technology: introducing an LRE special section

Cited by 24 publications

References 18 publications

Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field

Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field

Towards Replication in Computational Cognitive Modeling: a Machine Learning Perspective

Towards Replication in Computational Cognitive Modeling: A Machine Learning Perspective

Contact Info

Product

Resources

About