No System Is Perfect: Understanding How Registration‐Based Editorial Processes Affect Reproducibility and Investment in Research Quality

Bloomfield, Robert J.; Rennekamp, Kristina M.; Steenhoven, Blake A.

doi:10.1111/1475-679x.12208

Cited by 43 publications

(18 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Authors apparently expect readers to accept implicitly that such alterations have either not occurred or are appropriate. A new approach to reduce this problem is being explored that requires authors to describe their choices in advance of executing the research project and to communicate to the editors any changes thereafter (Bloomfield et al 2018;Kupferschmidt 2018). However, there is no assurance that this requirement will always be met, because the action may occur before initial submission.…”

Section: Model Modificationsmentioning

confidence: 99%

“…When this is the case, the ideal data set should be acknowledged and its absence justified, including any change in the variables selected and their measures. Unfortunately, while applauding the changes in the reviewing process championed by Bloomfield et al (2018) and others, we believe there is currently no way to assure that data tampering, including data mining, does not occur prior to the submission of a research paper to a journal.…”

Section: Testing the Modelmentioning

confidence: 99%

See 1 more Smart Citation

Important Issues in Statistical Testing and Recommended Improvements in Accounting Research

Dyckman

Zeff

2019

Econometrics

View full text Add to dashboard Cite

A great deal of the accounting research published in recent years has involved statistical tests. Our paper proposes improvements to both the quality and execution of such research. We address the following limitations in current research that appear to us to be ignored or used inappropriately: (1) unaddressed situational effects resulting from model limitations and what has been referred to as “data carpentry,” (2) limitations and alternatives to winsorizing, (3) necessary improvements to relying on a study’s calculated “p-values” instead of on the economic or behavioral importance of the results, and (4) the information loss incurred by under-valuing what can and cannot be learned from replications.

show abstract

Section: Model Modificationsmentioning

confidence: 99%

Section: Testing the Modelmentioning

confidence: 99%

Important Issues in Statistical Testing and Recommended Improvements in Accounting Research

Dyckman

Zeff

2019

Econometrics

View full text Add to dashboard Cite

show abstract

“…"The requirement … meant that the team was locked into using a research design that might not, in hindsight, be best suited to answer the research questions. In the traditional editorial process, researchers continue to refine their research design to suit the distributions of their data (Bloomfield, Rennekamp, and Steenhoven 2018) and also to take into consideration feedback provided during the workshopping of the paper. …[H]ence, led to difficulties in how the team most appropriately present the analysis to achieve the goal of conforming to the desired principles of responsible science".…”

Section: What Did Not Work?mentioning

confidence: 99%

Responsible science: Celebrating the 50-year legacy of using a registration-based framework

Aman

Beekes

Berkman

et al. 2019

Pacific-Basin Finance Journal

View full text Add to dashboard Cite

This paper represents the intersection of three spheres of influence, highly relevant to the global research community interested in a more reliable understanding of capital market phenomena. First, as a timely context, we celebrate the 50-year legacy of an iconic paper on event studies, Ball and Brown (1968). Second, we add our voice to the growing call for researchers to follow principles of "responsible science". Third, using the Ball and Brown paper as their inspiration, we report on an experiment in which several teams of researchers follow a registration-based editorial process, which illustrates one fruitful avenue on how more responsible research can be fostered in the future.

show abstract

“…It is an open question whether adopting Registered Reports results is more of an effort for the author or reviewer. Bloomfield et al (2018) analyzed papers submitted to a conference and concludes that the upfront 'investment' of the author(s) is increased, follow-up investment reduced. However, the overall effort to perform experiments seemed to exceed the typical level of effort in this (accounting) outlet.…”

Section: How Registered Reports Workmentioning

confidence: 99%

Introducing Registered Reports to the Information Systems Community

2019

View full text Add to dashboard Cite

Why We are Introducing Registered Reports Last February, the BISE Editorial Board met for its annual Editorial Board Meeting in Siegen with a bunch of new and interesting ideas on the agenda. One topic that was discussed and settled was the introduction of a new submission format-Registered Reports-for the recently established ''Human-Computer Interaction and Social Computing'' department. Experimental research is the dominant paradigm in this department and therefore a good candidate for this innovative approach to evaluating research results. Registered Reports are a promising format to encourage and support high quality (and also risky and innovative) experimental research and ensure rigorous scientific practice. As experimental research is gaining importance in our journal through our new department, but also in other areas such as Economics of IS, dealing with Registered Reports is also part of our strategy to maintain and develop our status as a high-quality community journal and make it even more attractive for potential authors in the respective areas. Within the last decade, the scientific landscape was shaken by a series of reports showing that many scientific studies are difficult or impossible to replicate or reproduce in a subsequent investigation, either by independent researchers or by the original researchers themselves. This problem is known as the ''Replication Crisis''. Though the focus was on psychology, biology and medicine, our domain-Information Systems-and related fields are no exception (Coiera et al. 2018; Head et al. 2015; Hutson 2018). Several questionable scientific practices-unfortunately so common that almost known by everyone-are more or less related to this problem (Chambers 2015; Chambers et al. 2014). One is a set of methods called ''phacking''. A common metaphorical paraphrase of p-hacking, somehow funny and sad at the same time, is ''torturing the data until they confess'' (e.g., Probst and Hagger 2015). P-hacking means, e.g., introducing or removing control variables or switching statistical tests in order to receive significant p-values. This is often combined with underpowered study design, the observation number of which is iteratively increased until the results match the expectations (very likely only due to variance). HARKing is another questionable method which consists in adapting hypotheses to the data after a study is performed. Simpson's paradox is the well-known phenomenon that a trend may appear in several different groups of data but disappears or changes to the contrary when these groups are combined. This illustrates that seemingly conflicting conclusions are possible for a given data set. These phenomena lead to the situation that a considerable amount of published findings are, in fact, false positives. Finally, a lack of data sharing makes many results not verifiable.

show abstract

No System Is Perfect: Understanding How Registration‐Based Editorial Processes Affect Reproducibility and Investment in Research Quality

Cited by 43 publications

References 38 publications

Important Issues in Statistical Testing and Recommended Improvements in Accounting Research

Important Issues in Statistical Testing and Recommended Improvements in Accounting Research

Responsible science: Celebrating the 50-year legacy of using a registration-based framework

Introducing Registered Reports to the Information Systems Community

Contact Info

Product

Resources

About