The experimental method is one of the staple methodological tools of the scientific method, and is as such prevalent in the psychological literature. It relies on the process of randomization to create equivalent groups. However, this procedure requires sufficiently large samples to succeed. In the current paper, we introduce tools that are based on the sampling distribution of Cohen’s d and that enable computing the likelihood that randomization succeeded in creating equivalent groups and the required sample size to achieve a desired likelihood of randomization success. The required sample sizes are considerable, and to illustrate this, we compute the likelihood of randomization failure using data from the Reproducability Project: Psychology. It is shown that it is likely that many original studies but also many replications failed to successfully create equivalent groups. For the replications, the mean likelihood of randomization failure was 44.54% (with a 95% confidence interval of [35.03%; 54.05%]) in the most liberal scenario, and 100% in the most conservative scenario. This means that many studies were in fact not experiments: the observed effects were at best conditional upon the value of unknown confounders, and at worst biased. In any case replication is unlikely when the randomization procedure failed to generate equivalent groups in either the original study or the replication. The consequence is that researchers in psychology, but also the funders of research in psychology, will have to get used to conducting considerably larger studies if they are to build a strong evidence base.