Single empirical tests are always ambiguous in their implications for the theory under investigation, because non-corroborative evidence leaves us underdetermined in our decision as to whether the main hypothesis or one or more auxiliary hypotheses should bear the burden of falsification. Methodological falsificationism tries to solve this problem by relegating auxiliary hypotheses that increase the testability of theories to unproblematic background knowledge and disallowing others. However, decisions to accept such auxiliaries as unproblematic are seldom conclusively justified in the social and behavioral sciences, where operationalizations play a central role, but are much less theory-driven and independently testable. Close and conceptual replications are crucial in tackling different aspects of underdetermination, but they fail to serve this purpose when conducted in isolation. To facilitate rational decision-making regarding falsifications, we propose Systematic Replications Framework (SRF) that organizes subsequent tests into a pre-planned series of logically interlinked close and conceptual replications. SRF reduces underdetermination by disentangling the implications of non-corroborative findings for the main hypothesis and the operationalization-related auxiliaries. It also serves as a severe-testing procedure through systematically organized self-replications. We further discuss how SRF will be particularly useful if applied to contested theoretical claims with mixed evidence and realized through adversarial collaboration. We argue that SRF will facilitate establishment of scientific consensus regarding established evidence and how exactly it supports or contradicts competing theoretical claims, which will in turn allow us to more objectively appraise scientific progress in the social and behavioral sciences.

Researchers commonly make dichotomous claims based on continuous test statistics. Many have branded the practice as misuse of statistics, and criticize scientists for suffering from “dichotomania”. However, the role dichotomous claims play in science is not primarily a statistical one, but an epistemological and pragmatic one. The epistemological function of dichotomous claims consists in transforming data into factual statements that can falsify a universal statement. This transformation requires pre-specified methodological decision procedures such as statistical hypothesis testing (e.g., Neyman-Pearson tests). From the perspective of methodological falsificationism these decision procedures are necessary, as probabilistic statements (e.g. continuous test statistics) cannot function as falsifiers of substantive hypotheses. However, they are not sufficient since for dichotomous claims to have any implication regarding theoretical claims about phenomena, there should be a valid derivation chain linking theoretical, experimental and data models. The pragmatic function of dichotomous claims is facilitating scrutiny and criticism among peers by generating contestable statements, a process referred to by Popper as 'conjectures and refutations', through which we can determine which theories withstand scrutiny the best. Abandoning dichotomous claims to combat the misuse of statistics would not improve scientific inferences but will sacrifice these crucial epistemic and pragmatic functions.

Falsificationist and confirmationist approaches provide two well-established ways of evaluating generalizability. Yarkoni rejects both and invents a third approach we call neo-operationalism. His proposal cannot work for the hypothetical concepts psychologists use, because the universe of operationalizations is impossible to define, and hypothetical concepts cannot be reduced to their operationalizations. We conclude that he is wrong in his generalizability-crisis diagnosis.