Quantifying evidential value and selective reporting in recent and 10-year past psychophysiological literature: A pre-registered P-curve analysis

Carbine, Kaylie A.; Lindsey, Hannah M.; Rodeback, Rebekah E.; Larson, Michael J.

doi:10.1016/j.ijpsycho.2019.06.004

Cited by 14 publications

(12 citation statements)

References 79 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These bias correction results stand in contrast to other bias-corrected meta-analyses in which effects became indistinguishable from zero after correction for publication bias (e.g., ego-depletion: Carter, Kofler, Forster, & McCullough, 2015). This positive news for the field coheres with recent metascientific investigations which indicated that studies published in three major psychophysiology journals (Psychophysiology; International Journal of Psychophysiology; Journal of Psychophysiology) showed good evidential value and relatively low evidence for selective reporting (Carbine, Lindsey, Rodeback, & Larson, 2019).…”

Section: Discussionsupporting

confidence: 53%

Assessing and adjusting for publication bias in the relationship between anxiety and the error-related negativity

Saunders

Inzlicht

2020

International Journal of Psychophysiology

View full text Add to dashboard Cite

Many clinical neuroscience investigations have suggested that trait anxiety is associated with increased neural reactivity to mistakes in the form of an event-related potential called the error-related negativity (ERN). Several recent meta-analyses indicated that the anxiety-ERN association was of a small-to-medium effect size, however, these prior investigations did not comprehensively adjust effect sizes for publication bias. Here, in an updated meta-analysis (k=58, N=3819), we found support for an uncorrected effect size of r =-.19, and applied a range of methods to test for and correct publication bias (trim-and-fill, PET, PEESE, Peters' test, threeparameter selection model). The majority of bias-correction methods suggested that the correlation between anxiety and the ERN is non-zero, but smaller than the uncorrected effect size (average adjusted effect size: r =-.12, range: r =-.05 to-.18). Moderation analyses also revealed more robust effects for clinical anxiety and anxious samples characterised by worry, however, it should be noted that these larger effects were also associated with elevated indicators of publication bias relative to the overall analysis. Mixed anxiety and sub-clinical anxiety were not associated with the amplitude of the ERN. Our results suggest that the anxiety-ERN relationship survives multiple corrections for publication bias, albeit not among all sub-types and populations of anxiety. Nevertheless, only 50% of the studies included in our analysis reported significant results, indicating that future research exploring the anxiety-ERN relationship would benefit from increased statistical power.

show abstract

Section: Discussionsupporting

confidence: 53%

Assessing and adjusting for publication bias in the relationship between anxiety and the error-related negativity

Saunders

Inzlicht

2020

International Journal of Psychophysiology

View full text Add to dashboard Cite

show abstract

“…For example, the statistical power to detect a medium size effect is estimated to be 44% in the fields of cognitive neuroscience and psychology (Szucs & Ioannidis, 2017). A preregistered analysis of psychophysiology studies using the p-curve approach showed relatively low average statistical power of recent studies (45%; Carbine, Lindsey, Rodeback, & Larson, 2019). Another preregistered review of 150 ERP studies estimated an achieved power of .35-.73 for medium effect sizes (Clayson, Carbine, Baldwin, & Larson, 2019).…”

mentioning

confidence: 99%

A registered report of error-related negativity and reward positivity as biomarkers of depression: P-Curving the evidence

Clayson

Carbine

Larson

2020

International Journal of Psychophysiology

Self Cite

View full text Add to dashboard Cite

Performance-monitoring event-related brain potentials (ERPs), such as the error-related negativity (ERN) and reward positivity (RewP), are advocated as biomarkers of depression symptoms and risk. However, a recent meta-analysis indicated effect size heterogeneity in the ERN and RewP literatures. Hence, advocating these ERPs as biomarkers of depression might be premature or possibly misguided due to the selective reporting of significant analyses on the part of researchers (e.g., p-hacking or omission of non-significant findings). The present study quantified the degree of selective reporting and the evidential value for a true relationship between depression and ERN and RewP using a p-curve analysis. We predicted that the ERN and RewP literatures would fail to show evidential value for a relationship between each ERP and depression. Contrary to expectations, both literatures showed evidential value, albeit weak.The statistical power of the included ERN studies was between 20% and 25%, and the statistical power of the RewP was around 27%. Taken together, these findings provide support for a relationship between these ERPs and depression, which strengthens claims that these ERPs represent candidate biomarkers of depression symptoms and risk. In light of the evidence for these relationships being weak, some recommendations moving forward include conducting a priori power analyses, increasing sample sizes to improve statistical power, assessing the internal consistency of ERP scores, and carefully planning statistical approaches to maximize power.Keywords: performance monitoring, depression, p curve, error-related negativity (ERN), reward positivity (RewP) 3 Performance-monitoring event-related brain potentials (ERPs), such as the error-related negativity (ERN) and reward positivity (RewP), are advocated as biomarkers or endophenotypes of depression symptoms and risk (e.g., Meyer, Bress, Hajcak, & Gibb, 2018;Nelson, Perlman, Klein, Kotov, & Hajcak, 2016). However, recent meta-analytic work indicates that both ERN and RewP literatures are replete with heterogenous effect sizes (Moran, Schroder, Kneip, & Moser, 2017). Although heterogeneous effect sizes might be due to known or unknown moderating factors, it is possible that heterogeneity is due to the selective reporting of significant analyses or undisclosed use of other questionable research practices (QRPs), and this possibility was not examined in the meta-analysis. If effect size heterogeneity is due to selective reporting/omission of non-significant findings or other QRPs, then advocating performancemonitoring ERPs as biomarkers of depression would be misguided or at least premature. The aim of the present registered report was to the quantify the degree of selective reporting and the evidential value for a true relationship between depression and ERN and RewP using a priori registered hypotheses and methods.

show abstract

“…In addition to the conventional methods discussed above, new methods such as p-curves for detecting phacking have emerged in recent years [36,37]. P-curves have been tested in various scientific disciplines [3,38,39], although no studies that we examined in the field of HSR have used this technique. The validity and usefulness of p-curves are subject to debate and accumulation of further empirical evidence [40][41][42][43].…”

Section: Discussionmentioning

confidence: 99%

Publication and related biases in health services research: a systematic review of empirical evidence

Ayorinde

Williams

Mannion

et al. 2020

BMC Med Res Methodol

View full text Add to dashboard Cite

Background: Publication and related biases (including publication bias, time-lag bias, outcome reporting bias and p-hacking) have been well documented in clinical research, but relatively little is known about their presence and extent in health services research (HSR). This paper aims to systematically review evidence concerning publication and related bias in quantitative HSR. Methods: Databases including MEDLINE, EMBASE, HMIC, CINAHL, Web of Science, Health Systems Evidence, Cochrane EPOC Review Group and several websites were searched to July 2018. Information was obtained from: (1) Methodological studies that set out to investigate publication and related biases in HSR; (2) Systematic reviews of HSR topics which examined such biases as part of the review process. Relevant information was extracted from included studies by one reviewer and checked by another. Studies were appraised according to commonly accepted scientific principles due to lack of suitable checklists. Data were synthesised narratively. Results: After screening 6155 citations, four methodological studies investigating publication bias in HSR and 184 systematic reviews of HSR topics (including three comparing published with unpublished evidence) were examined. Evidence suggestive of publication bias was reported in some of the methodological studies, but evidence presented was very weak, limited in both quality and scope. Reliable data on outcome reporting bias and p-hacking were scant. HSR systematic reviews in which published literature was compared with unpublished evidence found significant differences in the estimated intervention effects or association in some but not all cases. Conclusions: Methodological research on publication and related biases in HSR is sparse. Evidence from available literature suggests that such biases may exist in HSR but their scale and impact are difficult to estimate for various reasons discussed in this paper. Systematic review registration: PROSPERO 2016 CRD42016052333.

show abstract

Quantifying evidential value and selective reporting in recent and 10-year past psychophysiological literature: A pre-registered P-curve analysis

Cited by 14 publications

References 79 publications

Assessing and adjusting for publication bias in the relationship between anxiety and the error-related negativity

Assessing and adjusting for publication bias in the relationship between anxiety and the error-related negativity

A registered report of error-related negativity and reward positivity as biomarkers of depression: P-Curving the evidence

Publication and related biases in health services research: a systematic review of empirical evidence

Contact Info

Product

Resources

About