Assessing the Rate of Replication in Economics

Berry, James; Coffman, Lucas C.; Hanley, Douglas; Gihleb, Rania; Wilson, Alistair J.

doi:10.1257/aer.p20171119

Cited by 50 publications

(29 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The resulting collection of papers covered a range of topics including examination of the replication rate in economics (Berry et al, 2017); alternative initiatives to promote the replication rate in economics including the role of journals, the provision of data and code, and the adoption of ideas in other disciplines (Chang et al, 2017;Coffman et al, 2017;Höffler, 2017); meta analyses not being an alternative to replication (Anderson and Kichkha, 2017); the factors underlying the replication of studies, with Google scholar citation counts found a prominent issue (Sukhtankar, 2017).…”

Section: Replication In Economics and Other Disciplinesmentioning

confidence: 99%

Replicating rockets and feathers

Cook

Fosten

2019

Energy Economics

View full text Add to dashboard Cite

This paper revisits the literature of asymmetric adjustment in gasoline and diesel prices, also known as the 'rockets and feathers' hypothesis, to consider the issue of the replication of empirical research. We examine the notion of replication versus robustness proposed by Clemens (2017) and add to the literature with a further review of recent and historic work on replication in economics and other disciplines. We then focus on the rockets and feathers literature, finding that the majority of empirical work performs robustness checks rather than replication of earlier papers. We perform two contrasting replication case studies motivated by the ideas of misspecification analysis, dynamic specification, mark-up, pass-through and asymmetric adjustment. In the first case study we find that results are both replicable and robust, even when data specifications are not identical. However, in the second case study we find that the results using the original sample are overturned when reanalysing the problem using an improved model specification. Furthermore, when extending the sample with more recent data and using a more sophisticated method, asymmetry is detected in both petrol and diesel pricing; different to the findings of the original study. Particular care must be taken in future rockets and feathers replications with regard to model specification and methodology. JEL Classification: C22, C52, Q41

show abstract

Section: Replication In Economics and Other Disciplinesmentioning

confidence: 99%

Replicating rockets and feathers

Cook

Fosten

2019

Energy Economics

View full text Add to dashboard Cite

show abstract

“…A few recent papers attempt to address this issue and estimate the “natural rate of replication” in economics. First, Berry et al () focus on all the empirical papers published in the centenary volume (2010) of the American Economic Review , and manually code all their published citations as either replications, robustness tests, extensions, or none of the above. They find that less than a third of the 70 papers have been replicated at least once, where a replication is defined as a project “speaking directly to the veracity of the original paper.” Validating the assertion that the visibility of replications is low, Berry et al () find considerable uncertainty among the authors of the original papers over the number of extant replications of their studies.…”

Section: Dozen Thingsmentioning

confidence: 99%

“…First, Berry et al () focus on all the empirical papers published in the centenary volume (2010) of the American Economic Review , and manually code all their published citations as either replications, robustness tests, extensions, or none of the above. They find that less than a third of the 70 papers have been replicated at least once, where a replication is defined as a project “speaking directly to the veracity of the original paper.” Validating the assertion that the visibility of replications is low, Berry et al () find considerable uncertainty among the authors of the original papers over the number of extant replications of their studies. Second, Sukhtankar () analyzes 1056 empirical papers in development economics published in the top 10 general interest journals between 2000 and 2015, perform a reverse citation search, then search within the ensuing list for “replication” or alternative cognates.…”

Section: Dozen Thingsmentioning

confidence: 99%

The Dozen Things Experimental Economists Should Do (More of)

Czibor¹,

Jimenez-Gomez²,

List

2019

Southern Economic Journal

View full text Add to dashboard Cite

What was once broadly viewed as an impossibility—learning from experimental data in economics—has now become commonplace. Governmental bodies, think tanks, and corporations around the world employ teams of experimental researchers to answer their most pressing questions. For their part, in the past two decades academics have begun to more actively partner with organizations to generate data via field experimentation. Although this revolution in evidence‐based approaches has served to deepen the economic science, recently a credibility crisis has caused even the most ardent experimental proponents to pause. This study takes a step back from the burgeoning experimental literature and introduces 12 actions that might help to alleviate this credibility crisis and raise experimental economics to an even higher level. In this way, we view our “12 action wish list” as discussion points to enrich the field.

show abstract

“…According to Duvendack et al (2017) a broad definition of replication is a study that determines the validity of one or more empirical results from an original published study. Berry et al (2017) consider a replication to be "any project that reports results that speak directly to the veracity of the original paper's main hypothesis. "…”

Section: What Does Replication Entail?mentioning

confidence: 99%

Normalized CES supply-side system approach: how to replicate Klump, McAdam, and Willman (2007)

Daniels

Kakar

2018

Economics

View full text Add to dashboard Cite

This paper lays out a replication plan for the influential paper by Klump et al. (Factor Substitution and Factor-augmenting Technical Progress in the United States: a Normalized Supply-side System Approach 2007) on using a normalized CES supply-side system approach to estimate the value of the elasticity of substitution between factors of production and identify the growth patterns for biased technical progress. The authors begin with a general discussion of basic principles on carrying out a replication study. Further, they outline key steps to follow to replicate the chosen paper and establish criteria that can be used to determine if the replication confirms or disconfirms the original findings. This paper contributes to the increased interest in improving replications in economics research.

show abstract

Assessing the Rate of Replication in Economics

Cited by 50 publications

References 2 publications

Replicating rockets and feathers

Replicating rockets and feathers

The Dozen Things Experimental Economists Should Do (More of)

Normalized CES supply-side system approach: how to replicate Klump, McAdam, and Willman (2007)

Contact Info

Product

Resources

About