Metascience on Peer Review: Testing the Effects of a Study’s Originality and Statistical Significance in a Field Experiment

Elson, Malte; Huff, Markus; Utz, Sonja

doi:10.1177/2515245919895419

Cited by 17 publications

(9 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It remains largely unknown how much peer review improves research quality relative to other options (Elson et al, 2020) and whether the effect size of this improvement justifies the massive time costs, inefficiencies, and knowledge delays generated by the process. But given the pervasiveness of false-positive findings and contradictory claims, the behavioral sciences need more effective quality control procedures.…”

Section: The Goals Of Society and Sciencementioning

confidence: 99%

Keep your enemies close: Adversarial collaborations will improve behavioral science.

Clark¹,

Costello²,

Mitchell³

2022

Journal of Applied Research in Memory and Cognition

View full text Add to dashboard Cite

Behavioral scientists enjoy vast methodological freedom in how they operationalize theoretical constructs. This freedom may promote creativity in designing laboratory paradigms that shed light on real-world phenomena, but it also enables questionable research practices that undercut our collective credibility. Open Science norms impose some discipline but cannot constrain cherry-picking operational definitions that insulate preferred theories from rejection. All too often scholars conduct performative research to score points instead of engaging each other’s strongest arguments—a pattern that allows contradictory claims to fester unresolved for decades. Adversarial collaborations, which call on disputants to codevelop tests of competing hypotheses, are an efficient method of improving our science’s capacity for self-correction and of promoting intellectual competition that exposes false claims. Although individual researchers are often initially reluctant to participate, the research community would be better served by institutionalizing adversarial collaboration into its peer-review process.

show abstract

Section: The Goals Of Society and Sciencementioning

confidence: 99%

Keep your enemies close: Adversarial collaborations will improve behavioral science.

Clark¹,

Costello²,

Mitchell³

2022

Journal of Applied Research in Memory and Cognition

View full text Add to dashboard Cite

show abstract

“…Metascience is thus not to be confused with meta-analysis that summarizes effects in the literature on a certain research question quantitatively, but metascience means the science about the science and “ involves taking a bird’s eye view of science ” ( Ioannidis et al, 2015 ). The way we as HRI researchers conduct, report, verify, evaluate and incentivize research needs to be documented and analyzed in order to be corrected if needed (see Elson et al, 2020 as an example of a metascientific experiment).…”

Section: Toward More Reliable Hri Researchmentioning

confidence: 99%

Crisis Ahead? Why Human-Robot Interaction User Studies May Have Replicability Problems and Directions for Improvement

2022

View full text Add to dashboard Cite

There is a confidence crisis in many scientific disciplines, in particular disciplines researching human behavior, as many effects of original experiments have not been replicated successfully in large-scale replication studies. While human-robot interaction (HRI) is an interdisciplinary research field, the study of human behavior, cognition and emotion in HRI plays also a vital part. Are HRI user studies facing the same problems as other fields and if so, what can be done to overcome them? In this article, we first give a short overview of the replicability crisis in behavioral sciences and its causes. In a second step, we estimate the replicability of HRI user studies mainly 1) by structural comparison of HRI research processes and practices with those of other disciplines with replicability issues, 2) by systematically reviewing meta-analyses of HRI user studies to identify parameters that are known to affect replicability, and 3) by summarizing first replication studies in HRI as direct evidence. Our findings suggest that HRI user studies often exhibit the same problems that caused the replicability crisis in many behavioral sciences, such as small sample sizes, lack of theory, or missing information in reported data. In order to improve the stability of future HRI research, we propose some statistical, methodological and social reforms. This article aims to provide a basis for further discussion and a potential outline for improvements in the field.

show abstract

“…Just like evaluators in many other disciplines, HRI evaluators tend to favor papers that report positive results, i.e., papers with results supporting a hypothesis, as well as papers reporting novel studies. Thus, replication studies that fail to reproduce prior significant results are especially at a disadvantage regarding their evaluation as has been shown in experimental studies in real conference peer-review processes [ 28 ]. Regardless of the study being original or a replication, every research outcome that is based on rigorous scientific work is worthy of publication and indeed should be published.…”

Section: Bias In Hri Research Evaluation and Practicementioning

confidence: 99%

Reproducibility in Human-Robot Interaction: Furthering the Science of HRI

et al. 2022

View full text Add to dashboard Cite

Purpose of Review To discuss the current state of reproducibility of research in human-robot interaction (HRI), challenges specific to the field, and recommendations for how the community can support reproducibility. Recent Findings As in related fields such as artificial intelligence, robotics, and psychology, improving research reproducibility is key to the maturation of the body of scientific knowledge in the field of HRI. The ACM/IEEE International Conference on Human-Robot Interaction introduced a theme on Reproducibility of HRI to their technical program in 2020 to solicit papers presenting reproductions of prior research or artifacts supporting research reproducibility. Summary This review provides an introduction to the topic of research reproducibility for HRI and describes the state of the art in relation to the HRI 2020 Reproducibility theme. As a highly interdisciplinary field that involves work with technological artifacts, there are unique challenges to reproducibility in HRI. Biases in research evaluation and practice contribute to challenges in supporting reproducibility, and the training of researchers could be changed to encourage research reproduction. The authors propose a number of solutions for addressing these challenges that can serve as guidelines for the HRI community and related fields.

show abstract

Metascience on Peer Review: Testing the Effects of a Study’s Originality and Statistical Significance in a Field Experiment

Cited by 17 publications

References 41 publications

Keep your enemies close: Adversarial collaborations will improve behavioral science.

Keep your enemies close: Adversarial collaborations will improve behavioral science.

Crisis Ahead? Why Human-Robot Interaction User Studies May Have Replicability Problems and Directions for Improvement

Reproducibility in Human-Robot Interaction: Furthering the Science of HRI

Contact Info

Product

Resources

About