Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling

John, Leslie K.; Loewenstein, George; Prelec, Dražen

doi:10.1177/0956797611430953

Cited by 1,975 publications

(1,962 citation statements)

References 38 publications

Supporting

Mentioning

1,870

Contrasting

Unclassified

Order By: Relevance

“…This intuition is correct from an accuracy point of view: More data leads to more precise estimates (e.g., Maxwell, Kelley, & Rausch, 2008;Schönbrodt & Perugini, 2013). According to John, Loewenstein, and Prelec (2012), optionally increasing the sample size when the results are not significant is one of the most common (questionable) research practices. Furthermore, Yu, Sprenger, Thomas, and Dougherty (2013) showed empirically which 4 SCHÖNBRODT (incorrect) heuristics researchers used in their optional stopping practice.…”

Section: The Nhst Procedures With a Priori Power Analysis And Some Ofmentioning

confidence: 99%

Sequential hypothesis testing with Bayes factors: Efficiently testing mean differences.

Schönbrodt

Wagenmakers

Zehetleitner

et al. 2017

Psychological Methods

438

466

View full text Add to dashboard Cite

Unplanned optional stopping rules have been criticized for inflating Type I error rates under the null hypothesis significance testing (NHST) paradigm. Despite these criticisms this research practice is not uncommon, probably as it appeals to researcher's intuition to collect more data in order to push an indecisive result into a decisive region. In this contribution we investigate the properties of a procedure for Bayesian hypothesis testing that allows optional stopping with unlimited multiple testing, even after each participant. In this procedure, which we call Sequential Bayes Factors (SBF), Bayes factors are computed until an a priori defined level of evidence is reached. This allows flexible sampling plans and is not dependent upon correct e↵ect size guesses in an a priori power analysis. We investigated the long-term rate of misleading evidence, the average expected sample sizes, and the biasedness of e↵ect size estimates when an SBF design is applied to a test of mean di↵erences between two groups. Compared to optimal NHST, the SBF design typically needs 50% to 70% smaller samples to reach a conclusion about the presence of an e↵ect, while having the same or lower long-term rate of wrong inference.

show abstract

Section: The Nhst Procedures With a Priori Power Analysis And Some Ofmentioning

confidence: 99%

Sequential hypothesis testing with Bayes factors: Efficiently testing mean differences.

Schönbrodt

Wagenmakers

Zehetleitner

et al. 2017

Psychological Methods

438

466

View full text Add to dashboard Cite

show abstract

“…The correct way to incorporate the results of data collected after the analysis of an initial tranche of data is via meta-analysis (Braver et al 2014). Just adding the new data to the existing data set is wrong, since it involves deciding to collect more data after looking at the results (John et al 2012). Equally, meta-analysis of all six studies is not a valid approach because the five studies in the first tranche were planned in advance (before the experiment) as defined in the protocol.…”

Section: Phase 2 Analysismentioning

confidence: 99%

Robust Statistical Methods for Empirical Software Engineering

et al. 2016

View full text Add to dashboard Cite

There have been many changes in statistical theory in the past 30 years, including increased evidence that non-robust methods may fail to detect important results. The statistical advice available to software engineering researchers needs to be updated to address these issues. This paper aims both to explain the new results in the area of robust analysis methods and to provide a large-scale worked example of the new methods. We summarise the results of analyses of the Type 1 error efficiency and power of standard parametric and non-parametric statistical tests when applied to non-normal data sets. We identify parametric and non-parametric methods that are robust to non-normality. We present an analysis of a large-scale software engineering experiment to illustrate their use. We illustrate the use of kernel density plots, and parametric and non-parametric methods using four different software engineering data sets. We explain why the methods are necessary and the rationale for selecting a specific analysis. We suggest using kernel density plots rather than box plots to visualise data distributions. For parametric analysis, we recommend trimmed means, which can support reliable tests of the differences between the central location of two or

show abstract

“…However, questionable research practices such as manipulating statistical analyses until desired results are produced (i.e., data fishing and p-hacking; see https://projects.fivethirtyeight.com/p-hacking/ for an interactive demonstration), hypothesizing after results are known (i.e., HARKing), and selectively reporting analyses and publishing studies with positive results (i.e., reporting and publication bias) are not so clearly proscribed. Researchers report engaging in these questionable research practices, PROMOTING OPEN SCIENCE 8 which can affect study findings and influence the trustworthiness of evidence (John et al, 2012;Simmons et al, 2011).…”

Section: Concerns About the Culture Of Contemporary Sciencementioning

confidence: 99%

Promoting Open Science to Increase the Trustworthiness of Evidence in Special Education

Cook

Lloyd

Mellor

et al. 2018

Exceptional Children

100

View full text Add to dashboard Cite

Scientific evidence should guide the selection of practice for individuals with disabilities. Scientific evidence, however, must be trustworthy to move special education toward greater empirical certainty and more effective policies and practices. Transparency, openness, and reproducibility increase the trustworthiness of evidence. We propose that researchers in special education adopt emerging open-science reforms, such as preprints, data and materials sharing, preregistration of studies and analysis plans, and Registered Reports. Adoption of these practices will require shifts in cultural norms, guidelines, and incentives. We discuss how adopting open-science practices can advance the quality of research and, consequently, policy and practice in special education.

show abstract

Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling

Cited by 1,975 publications

References 38 publications

Sequential hypothesis testing with Bayes factors: Efficiently testing mean differences.

Sequential hypothesis testing with Bayes factors: Efficiently testing mean differences.

Robust Statistical Methods for Empirical Software Engineering

Promoting Open Science to Increase the Trustworthiness of Evidence in Special Education

Contact Info

Product

Resources

About