Gangestad et al. (this issue) recently published alternative analyses of our open data to investigate whether women show ovulatory shifts in preferences for men’s bodies. They argue that a significant three-way interaction between log-transformed hormones, a muscularity component, and women’s relationship status provides evidence for the ovulatory shift hypothesis. Their conclusion is opposite to the one we previously reported (Jünger et al., 2018). Here, we provide evidence that Gangestad et al.’s differing conclusions are contaminated by overfitting, clarify reasons for deviating from our preregistration in some aspects, discuss the implications of data-dependent re-analysis, and report a multiverse analysis which provides evidence that their reported results are not robust. Further, we use the current debate to contrast the risk of prematurely concluding a null effect against the risk of shielding hypotheses from falsification. Finally, we discuss the benefits and challenges of open scientific practices, as contested by Gangestad et al., and conclude with implications for future studies.