A Systematic Review of Reproducibility Research in Natural Language Processing

Abstract

Against the background of what has been termed a reproducibility crisis in science, the NLP field is becoming increasingly interested in, and conscientious about, the reproducibility of its results. The past few years have seen an impressive range of new initiatives, events and active research in the area. However, the field is far from reaching a consensus about how reproducibility should be defined, measured and addressed, with diversity of views currently increasing rather than converging. With this focused contribution, we aim to provide a wideangle, and as near as possible complete, snapshot of current work on reproducibility in NLP, delineating differences and similarities, and providing pointers to common denominators.

Is Stance Detection Topic-Independent and Cross-topic Generalizable? -- A Reproduction Study

“…Our contributions are threefold. We first complete a reproduction of state-of-the-art cross-topic stance detection work (Reimers et al, 2019), as reproduction has repeatedly shown to be relevant for NLP (Fokkens et al, 2013;Cohen et al, 2018;Belz et al, 2021). The reproduction is largely successful: we obtain similar numeric results.…”

“…We adopt the definition of reproduction by Belz et al (2021): repeating the experiments as described in the earlier study, with the exact same data and software. We analyze our reproduced results according to the three dimensions of repro-duction proposed by Cohen et al ( 2018): whether we find either the same or different (1) (numeric) values, (2) findings, and (3) conclusions as the earlier study.…”

A Review of Human Evaluation for Style Transfer

“…In addition, raw annotations can shed light on the difficulty of the task and nature of the data: they can be aggregated in multiple ways (Oortwijn et al, 2021), or used to account for annotator bias in model training (Beigman and Beigman Klebanov, 2009). Finally, releasing annotated judgments makes it possible to replicate and further analyze the evaluation outcome (Belz et al, 2021).…”

“…For ST, the lack of detail and clarity in describing evaluation protocols makes it difficult to improve them, as has been pointed out for other NLG tasks by Shimorina and Belz (2021) who propose evaluation datasheets for clear documentation of human evaluations, Lee (2020) and van der who propose best practices guidelines, and Belz et al ( , 2021 who raise concerns regarding reproducibility. This issue is particularly salient for ST tasks where stylistic changes are defined implicitly by data (Jin et al, 2021) and where the instructions given to human judges for style transfer might be the only explicit characterization of the style dimension targeted.…”

“…Inspired by recent critiques of human evaluations of Natural Language Generation (NLG) systems Lee, 2020;Belz et al, , 2021Shimorina and Belz, 2021), we conduct a structured review of human evaluation for neural style transfer systems as their evaluation is primarily based on human judgments. Concretely, out of the 97 papers we reviewed, 69 of them resort to human evaluation (Figure 1), where it is treated either as a substitute for automatic metrics or as a more reliable evaluation.…”