Do Research Literatures Give Correct Answers?

Howard, George S.; Lau, Michael Y.; Maxwell, Scott E.; Venter, André; Lundy, Rae; Sweeny, Ryan M.

doi:10.1037/a0015468

Cited by 19 publications

(23 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Potentially problematic practices include selective reporting, selective analysis, and insufficient specification of the conditions necessary or sufficient to obtain the results (12)(13)(14)(15)(16)(17)(18)(19)(20)(21)(22)(23). We were inspired to address the gap in direct empirical evidence about reproducibility.…”

Section: Resultsmentioning

confidence: 98%

See 1 more Smart Citation

Estimating the reproducibility of psychological science

Aarts¹,

Anderson

et al. 2015

Science

5,686

2,020

View full text Add to dashboard Cite

52% Yes, a signiicant crisis 3% No, there is no crisis 7% Don't know 38% Yes, a slight crisis 38% Yes, a slight crisis 1,576 RESEARCHERS SURVEYED M ore than 70% of researchers have tried and failed to reproduce another scientist's experiments, and more than half have failed to reproduce their own experiments. Those are some of the telling figures that emerged from Nature's survey of 1,576 researchers who took a brief online questionnaire on reproducibility in research. The data reveal sometimes-contradictory attitudes towards reproduc-ibility. Although 52% of those surveyed agree that there is a significant 'crisis' of reproducibility, less than 31% think that failure to reproduce published results means that the result is probably wrong, and most say that they still trust the published literature. Data on how much of the scientific literature is reproducible are rare and generally bleak. The best-known analyses, from psychology 1 and cancer biology 2 , found rates of around 40% and 10%, respectively. Our survey respondents were more optimistic: 73% said that they think that at least half of the papers in their field can be trusted, with physicists and chemists generally showing the most confidence. The results capture a confusing snapshot of attitudes around these issues, says Arturo Casadevall, a microbiologist at the Johns Hopkins Bloomberg School of Public Health in Baltimore, Maryland. "At the current time there is no consensus on what reproducibility is or should be. " But just recognizing that is a step forward, he says. "The next step may be identifying what is the problem and to get a consensus. "

show abstract

Section: Resultsmentioning

confidence: 98%

“…Concerns about reproducibility are widespread across disciplines (9)(10)(11)(12)(13)(14)(15)(16)(17)(18)(19)(20)(21). Reproducibility is not well understood because the incentives for individual scientists prioritize novelty over replication (20).…”

Section: Implications and Limitationsmentioning

confidence: 99%

Estimating the reproducibility of psychological science

Aarts¹,

Anderson

et al. 2015

Science

5,686

2,020

View full text Add to dashboard Cite

show abstract

“…Several factors relating to the replication crisis have recently been examined, including the arbitrary inclusion of covariates and selective adjustment of sample size (Simmons, Nelson, & Simonsohn, 2011), the different and at times improper ways of handling outliers (Bakker & Wicherts, 2014), and the pervasive “file-drawer” problem affecting the field as a whole (Howard et al, 2009). The importance of addressing the replication crisis is evident.…”

mentioning

confidence: 99%

Inference Based on the Best-Fitting Model Can Contribute to the Replication Crisis: Assessing Model Selection Uncertainty Using a Bootstrap Approach

Lubke

Campbell

2016

Structural Equation Modeling: A Multidisciplinary Journal

View full text Add to dashboard Cite

Inference and conclusions drawn from model fitting analyses are commonly based on a single “best-fitting” model. If model selection and inference are carried out using the same data model selection uncertainty is ignored. We illustrate the Type I error inflation that can result from using the same data for model selection and inference, and we then propose a simple bootstrap based approach to quantify model selection uncertainty in terms of model selection rates. A selection rate can be interpreted as an estimate of the replication probability of a fitted model. The benefits of bootstrapping model selection uncertainty is demonstrated in a growth mixture analyses of data from the National Longitudinal Study of Youth, and a 2-group measurement invariance analysis of the Holzinger-Swineford data.

show abstract

“…Furthermore, Kromrey and Rendina-Gobioff [41] concluded that current statistical methods to account for publication bias may fail to control Type I error rates or lack sufficient power. Therefore, Howard et al [42] proposed a different way of addressing the file drawer problem using the example of the psychotherapy efficacy literature. Rather than correcting for bias statistically, they suggested performing a new mini-literature review meta-analysis of all their new studies and whether the results approached the value of a meta-analysis obtained from the entire …”

Section: Publication Bias: the "File Drawer Problem"mentioning

confidence: 99%

Meta-Analysis of Psychopharmacologic Treatment of Child and Adolescent Depression: Deconstructing Previous Reviews, Moving Forward

Maag¹

2014

J Psychol Psychother

View full text Add to dashboard Cite

The proliferation of Selective Serotonin Reuptake Inhibitors (SSRIs) in the 1980s led to increased use of antidepressant medications for children and adolescents with Major Depressive Disorder (MDD). Since then, there have been 18 reviews of this literature with nine being meta-analytic. Many of these meta-analyses suffer from several methodological problems: did not statistically compare medication efficacy, included only randomized placebo control trials, calculated response rate rather than risk-difference and odds ratio, were conducted prior to the 2009 publication of the PRISMA meta-analyses standards, rarely addressed publication bias, and failed to conduct metaregressions to account for moderator variables. The purpose of the present meta-analysis was to address each of these limitations. Results indicated SSRIs were the most effective class of medication with Sertraline having the highest response rate and Citalopram having the lowest response rate. Overall, Nefazodone had the highest response rate of any medication regardless of class, although there was a relatively small sample size (n = 39). When examining publication bias, only SSRIs had statistically significant positive findings. In terms of moderator variables, RCTs and open-label trials predicted response rate, as did age and gender (females).

show abstract

Do Research Literatures Give Correct Answers?

Cited by 19 publications

References 15 publications

Estimating the reproducibility of psychological science

Estimating the reproducibility of psychological science

Inference Based on the Best-Fitting Model Can Contribute to the Replication Crisis: Assessing Model Selection Uncertainty Using a Bootstrap Approach

Meta-Analysis of Psychopharmacologic Treatment of Child and Adolescent Depression: Deconstructing Previous Reviews, Moving Forward

Contact Info

Product

Resources

About