Big little lies: a compendium and simulation of<i>p</i>-hacking strategies

Stefan, Angelika; Schönbrodt, Felix D.

doi:10.1098/rsos.220346

Cited by 55 publications

(45 citation statements)

References 141 publications

(353 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Moreover, historically the eld has suffered from publication biases, in that studies that nd signi cant evidence for an effect have tended to be far more likely to be published in comparison to those that fail to nd signi cant evidence (Greenwald, 1975;Rosenthal, 1979). Given how common these questionable research practices seem to be (John et al, 2012;Stefan & Schönbrodt, 2023), they not only raise questions about the evidential value of a single study, but about the evidential value of an entire literature. But how can researchers best assess the evidential value of an expansive psychological literature?…”

Section: Full Textmentioning

confidence: 99%

Managing the terror of publication bias: A comprehensive p-curve analysis of the Terror Management Theory literature

Chen

Benjamin

Guo

et al. 2023

Preprint

View full text Add to dashboard Cite

We assessed the evidential value of the large literature (k=826 studies) investigating the mortality salience (MS) hypothesis from terror management theory. We employed a multitool assessment approach and reviewed past efforts to replicate experiments testing the MS hypothesis, and conducted a p-curve, a z-curve, and a random effects meta-analysis including bias corrections of the selection model, PET-PEESE, and WAAP-WLS on the studies. Overall, the different meta-analytic tools pointed to conflicting conclusions, reflecting differences in the methodology and philosophy of these tools. Our synthesis of these findings suggests there are true effects underlying some studies of the MS hypothesis, although the effects are highly heterogeneous, and the majority of studies are underpowered. We recommend future replications to assume a smaller effect size (r = .10 ~ .15) and to follow expert guidance in the experimental protocol. Given the conflicting findings that emerged, we suggest that future attempts to evaluate other literatures would benefit from a multitool assessment approach.

show abstract

Section: Full Textmentioning

confidence: 99%

Managing the terror of publication bias: A comprehensive p-curve analysis of the Terror Management Theory literature

Chen

Benjamin

Guo

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…The type of

p

‐hacking we simulated could arise in practice if, for example, investigators fit models to disjoint subsets of the data 32 . Many other forms of

p

‐hacking are possible 33,75 ; we simply chose one mechanism for this non‐comprehensive simulation study.…”

Section: Simulation Studymentioning

confidence: 99%

Sensitivity analysis for the interactive effects of internal bias and publication bias in meta‐analyses

Mathur

2023

Research Synthesis Methods

View full text Add to dashboard Cite

Meta‐analyses can be compromised by studies' internal biases (e.g., confounding in nonrandomized studies) as well as publication bias. These biases often operate nonadditively: publication bias that favors significant, positive results selects indirectly for studies with more internal bias. We propose sensitivity analyses that address two questions: (1) “For a given severity of internal bias across studies and of publication bias, how much could the results change?”; and (2) “For a given severity of publication bias, how severe would internal bias have to be, hypothetically, to attenuate the results to the null or by a given amount?” These methods consider the average internal bias across studies, obviating specifying the bias in each study individually. The analyst can assume that internal bias affects all studies, or alternatively that it only affects a known subset (e.g., nonrandomized studies). The internal bias can be of unknown origin or, for certain types of bias in causal estimates, can be bounded analytically. The analyst can specify the severity of publication bias or, alternatively, consider a “worst‐case” form of publication bias. Robust estimation methods accommodate non‐normal effects, small meta‐analyses, and clustered estimates. As we illustrate by re‐analyzing published meta‐analyses, the methods can provide insights that are not captured by simply considering each bias in turn. An R package implementing the methods is available (multibiasmeta).

show abstract

“…23 Given this diversity of p-hacking mechanisms, they called for future research to develop mathematical models of p-hacking strategies. 23 A key goal of the present article is to provide one possible formalization of p-hacking, enabling new conceptual insights about when and how p-hacking produces bias and how p-hacking relates conceptually to classical publication bias. This formalization additionally enables us to develop new analysis methods.…”

Section: Introductionmentioning

confidence: 99%

P‐hacking in meta‐analyses: A formalization and new meta‐analytic methods

Mathur

2024

Research Synthesis Methods

View full text Add to dashboard Cite

As traditionally conceived, publication bias arises from selection operating on a collection of individually unbiased estimates. A canonical form of such selection across studies (SAS) is the preferential publication of affirmative studies (i.e., those with significant, positive estimates) versus nonaffirmative studies (i.e., those with nonsignificant or negative estimates). However, meta‐analyses can also be compromised by selection within studies (SWS), in which investigators “p‐hack” results within their study to obtain an affirmative estimate. Published estimates can then be biased even conditional on affirmative status, which comprises the performance of existing methods that only consider SAS. We propose two new analysis methods that accommodate joint SAS and SWS; both analyze only the published nonaffirmative estimates. First, we propose estimating the underlying meta‐analytic mean by fitting “right‐truncated meta‐analysis” (RTMA) to the published nonaffirmative estimates. This method essentially imputes the entire underlying distribution of population effects. Second, we propose conducting a standard meta‐analysis of only the nonaffirmative studies (MAN); this estimate is conservative (negatively biased) under weakened assumptions. We provide an R package (phacking) and website (metabias.io). Our proposed methods supplement existing methods by assessing the robustness of meta‐analyses to joint SAS and SWS.

show abstract

Big little lies: a compendium and simulation ofp-hacking strategies

Cited by 55 publications

References 141 publications

Managing the terror of publication bias: A comprehensive p-curve analysis of the Terror Management Theory literature

Managing the terror of publication bias: A comprehensive p-curve analysis of the Terror Management Theory literature

Sensitivity analysis for the interactive effects of internal bias and publication bias in meta‐analyses

P‐hacking in meta‐analyses: A formalization and new meta‐analytic methods

Contact Info

Product

Resources

About