An Evaluation of Four Solutions to the Forking Paths Problem: Adjusted Alpha, Preregistration, Sensitivity Analyses, and Abandoning the Neyman-Pearson Approach

Abstract

Gelman and Loken (2013, 2014) proposed that when researchers base their statistical analyses on the idiosyncratic characteristics of a specific sample (e.g., a nonlinear transformation of a variable because it is skewed), they open up alternative analysis paths in potential replications of their study that are based on different samples (i.e., no transformation of the variable because it is not skewed). These alternative analysis paths count as additional (multiple) tests and, consequently, they increase the probability of making a Type I error during hypothesis testing. The present article considers this forking paths problem and evaluates four potential solutions that might be used in psychology and other fields: (a) adjusting the prespecified alpha level, (b) preregistration, (c) sensitivity analyses, and (d) abandoning the Neyman-Pearson approach. It is concluded that although preregistration and sensitivity analyses are effective solutions to p-hacking, they are ineffective against result-neutral forking paths, such as those caused by transforming data. Conversely, although adjusting the alpha level cannot address p-hacking, it can be effective for result-neutral forking paths. Finally, abandoning the Neyman-Pearson approach represents a further solution to the forking paths problem.Keywords: forking paths; null hypothesis significance testing; preregistration; replication crisis; sensitivity analyses Copyright © 2017, American Psychological Association. This self-archived article is provided for non-commercial and scholarly purposes only. Forking Paths 2 Imagine that a researcher predicts that men have higher self-esteem than women. Following data collection, the researcher observes that self-esteem scores in the sample are negatively skewed and, therefore, not ideal for a parametric test. Consequently, the researcher decides to log10 transform the selfesteem scores in order to produce a more normal distribution. Although this transformation makes the selfesteem scores more suitable for a t test, it also causes a statistical problem that increases the probability of incorrectly rejecting the null hypothesis that there is no gender difference in self-esteem (i.e., making a Type I error).This example represents part of a broader problem that Gelman and Loken (2013, 2014) have described as the garden of forking paths. Forking paths represent tests that are based on different analytical approaches, such as transforming or not transforming self-esteem scores. In the above example, the researcher followed only one analysis path (transform the scores) because it was considered to be the best (most valid) path to take. Nonetheless, the fact that a second potential analysis path exists (do not transform the scores) increases the probability of making a false positive (Type I) error during hypothesis testing.In their articles on this subject, Loken (2013, 2014) provided some useful illustrations of the forking paths problem using examples from several published research articles. However, they did not pro...

Which adolescent factors predict alcohol misuse in young adulthood? A co‐twin comparisons study

“…We adopted a P ‐value threshold of 0.05, given that our directional hypotheses and analytical plan were pre‐registered, which is consistent with Nosek et al . and Rubin .…”

Does preregistration improve the credibility of research findings?

“…However, the familywise error rate for the entire set of tests in an exploratory data analysis is only relevant if researchers are interested in testing a joint null hypothesis that may be rejected following at least one significant result in this analysis. In practice, researchers are unlikely to be interested in this studywise error rate, because the associated studywise hypothesis is not likely to be theoretically meaningful (Rubin, 2017a(Rubin, , 2017b(Rubin, , 2019a.…”

Improving practices and inferences in developmental cognitive neuroscience

“…Since the research plan is transparently disseminated prior to conducting research, those interpreting the results can be assured that the outcomes of these clinical trials, based on the NHST principles discussed above, are correctly conditioned on the number of comparisons performed. This strategy is one of several possible solutions to the problem of forking paths ( Rubin, 2017 ). It works well when the correct analytic plan is known and can be specified a priori, and when the data are unlikely to deviate in surprising ways from the assumptions of that plan.…”