Effect Size Estimation From<i>t</i>-Statistics in the Presence of Publication Bias

Ulrich, Rolf; Miller, Jeff; Erdfelder, Edgar

doi:10.1027/2151-2604/a000319

Cited by 25 publications

(21 citation statements)

References 99 publications

(285 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…At the same time, the proportion of significant results among published studies increased to 0.22 / (0.22 + 0.78 × 0.10) = 73.8%. As a consequence, significant studies had a stronger weight on mean estimates, and the resulting bias was larger than with δ = 0 ( = 0.41; a formal analysis of the relationship between different selection mechanisms and the resulting bias in effect size estimates is given in Ulrich, Miller & Erdfelder, 2018). Carter & Doucouliagos, 2018).…”

Section: Bias In Meta-analytic Effect Size Estimatesmentioning

confidence: 99%

How to detect publication bias in psychological research? A comparative evaluation of six statistical methods

Renkewitz¹,

Keiner²

2018

Preprint

View full text Add to dashboard Cite

Publication biases and questionable research practices are assumed to be two of the main causes of low replication rates observed in the social sciences. Both of these problems do not only increase the proportion of false positives in the literature but can also lead to severely inflated effect size estimates in meta-analyses. Methodologists have proposed a number of statistical tools to detect and correct such bias in meta-analytic results. We present an evaluation of the performance of six of these tools in detecting bias. To assess the Type I error rate and the statistical power of these tools we simulated a large variety of literatures that differed with regard to underlying true effect size, heterogeneity, number of available primary studies and variation of sample sizes in these primary studies. Furthermore, simulated primary studies were subjected to different degrees of publication bias. Our results show that the power of the detection methods follows a complex pattern. Across all simulated conditions, no method consistently outperformed all others. Hence, choosing an optimal method would require knowledge about parameters (e.g., true effect size, heterogeneity) that meta-analysts cannot have. Additionally, all methods performed badly when true effect sizes were heterogeneous or primary studies had a small chance of being published irrespective of their results. This suggests, that in many actual meta-analyses in psychology bias will remain undiscovered no matter which detection method is used.

show abstract

Section: Bias In Meta-analytic Effect Size Estimatesmentioning

confidence: 99%

How to detect publication bias in psychological research? A comparative evaluation of six statistical methods

Renkewitz¹,

Keiner²

2018

Preprint

View full text Add to dashboard Cite

show abstract

“…In the presence of such publication bias, true effect sizes tend to be overestimated, especially when the true effects and sample sizes are small (Hedges, 1984). Special statistical techniques may be helpful in removing the positive estimation bias (e.g., Hedges, 1992;Iyengar & Greenhouse, 1988;Simonsohn et al, 2014a;Ulrich, Miller, & Erdfelder, 2016), but effect sizes could certainly be estimated more accurately if the observed effects were not restricted by publication bias in the first place. Determining the appropriate individual outcome payoffs is also crucial for finding the optimal values of sample size and α within a scenario, but the individual payoff values are particularly difficult to determine because they are somewhat subjective.…”

Section: Critical Unknowns: Base Rates Effect Sizes and Individual mentioning

confidence: 99%

Optimizing Research Payoff

Miller

Ulrich

2016

Perspect Psychol Sci

Self Cite

View full text Add to dashboard Cite

In this article, we present a model for determining how total research payoff depends on researchers' choices of sample sizes, α levels, and other parameters of the research process. The model can be used to quantify various trade-offs inherent in the research process and thus to balance competing goals, such as (a) maximizing both the number of studies carried out and also the statistical power of each study, (b) minimizing the rates of both false positive and false negative findings, and (c) maximizing both replicability and research efficiency. Given certain necessary information about a research area, the model can be used to determine the optimal values of sample size, statistical power, rate of false positives, rate of false negatives, and replicability, such that overall research payoff is maximized. More specifically, the model shows how the optimal values of these quantities depend upon the size and frequency of true effects within the area, as well as the individual payoffs associated with particular study outcomes. The model is particularly relevant within current discussions of how to optimize the productivity of scientific research, because it shows which aspects of a research area must be considered and how these aspects combine to determine total research payoff.

show abstract

“…Another major motivation for providing a compilation of the so far published RT studies by means of a meta‐analysis is to assess the size of the space–time congruency effect in those experiments, in which the concept of time is made salient, as there is further substantial variation concerning the design of the conducted studies beyond the salience of time. Under these circumstances a potential publication bias—that is, nonsignificant or negative results are put into the researcher's file drawer and do not get published—could imply that the actual space–time congruency effect is smaller than portrayed by the published studies and hence might not even significantly deviate from zero (Rosenthal, ; Ulrich, Miller, & Erdfelder, ). We will incorporate the potential publication bias into the estimation of the real effect size in order to examine whether it significantly deviates from zero and, thus, can be considered a sound effect.…”

Section: Introductionmentioning

confidence: 99%

The Space–Time Congruency Effect: A Meta‐Analysis

et al. 2019

Self Cite

View full text Add to dashboard Cite

Several reaction time (RT) studies report faster responses when responses to temporal information are arranged in a spatially congruent manner than when this arrangement is incongruent. The resulting space–time congruency effect is commonly attributed to a culturally salient localization of temporal information along a mental timeline (e.g., a mental timeline that runs from left to right). The present study aims to provide a compilation of the published RT studies on this time–space association in order to estimate the size of its effect and the extent of potential publication bias in this field of research. In this meta‐analysis, three types of task are distinguished due to hitherto existing empirical findings. These findings suggest that the extent to which time is made relevant to the experimental task has a systematic impact on whether or not the mental timeline is activated. The results of this meta‐analysis corroborate these considerations: First, experiments that make time a task‐relevant dimension have a mean effect size of d = 0.46. Second, in experiments in which time is task irrelevant, the effect size does not significantly deviate from zero. Third, temporal priming studies have a surprisingly high mean effect size of d = 0.47, which, however, should be adjusted to d = 0.36 due to publication bias.

show abstract

Effect Size Estimation Fromt-Statistics in the Presence of Publication Bias

Cited by 25 publications

References 99 publications

How to detect publication bias in psychological research? A comparative evaluation of six statistical methods

How to detect publication bias in psychological research? A comparative evaluation of six statistical methods

Optimizing Research Payoff

The Space–Time Congruency Effect: A Meta‐Analysis

Contact Info

Product

Resources

About