When data analysts operate within different statistical frameworks (e.g., frequentist versus Bayesian, emphasis on estimation versus emphasis on testing), how does this impact the qualitative conclusions that are drawn for real data? To study this question empirically we selected from the literature two simple scenarios --involving a comparison of two proportions and a Pearson correlation-- and asked four teams of statisticians to provide a concise analysis and a qualitative interpretation of the outcome. The results showed considerable overall agreement; nevertheless, this agreement did not appear to diminish the intensity of the subsequent debate over which statistical framework is more appropriate to address the questions at hand.

We present consensus-based guidance for conducting and documenting multi-analyst studies. We discuss why broader adoption of the multi-analyst approach will strengthen the robustness of results and conclusions in empirical sciences.

Scientific theories reflect some of humanity's greatest epistemic achievements. The best theories motivate us to search for discoveries, guide us towards successful interventions, and help us to explain and organize knowledge. Such theories require a high degree of specificity, and specifying them requires modeling skills. Unfortunately, in psychological science, theories are often not precise, and psychological scientists often lack the technical skills to formally specify existing theories. This problem raises the question: How can we promote formal theory development in psychology, where there are many content experts but few modelers? In this paper, we discuss one strategy for addressing this issue: a Many Modelers approach. Many Modelers consist of mixed teams of modelers and non-modelers that collaborate to create a formal theory of a phenomenon. We report a proof of concept of this approach, which we piloted as a three-hour hackathon at the SIPS 2021 conference. We find that (a) psychologists who have never developed a formal model can become excited about formal modeling and theorizing; (b) a division of labor in formal theorizing could be possible where only one or a few team members possess the prerequisite modeling expertise; and (c) first working prototypes of a theoretical model can be created in a short period of time.

How confident are researchers in their own claims? Augustus De Morgan suggested that researchers may initially present their conclusions modestly, but afterwards use them as if they were a “moral certainty”. To prevent this from happening, De Morgan proposed that whenever researchers make a claim, they accompany it with a number that reflects their degree of confidence. Current reporting procedures in academia, however, usually present claims without the authors’ assessment of confidence. Here, we report the partial results from an anonymous questionnaire on the concept of evidence.

Just as teachers give students exams to assess their mastery of a subject, researchers submit their theories to empirical tests. And just as a high score on a test by itself is not sufficient to believe in the student’s mastery of a subject, researchers need severe tests to make reliable inferences from observations to theories. In this paper, we provide an explication of the concept of severity, and how it underlies three current methodological crises in psychology: the theory crisis, the measurement crisis, and the generalizability crisis. Our detailed account reinforces the importance of designing tests that can prove yourself wrong, and should assist empirical researchers in evaluating the severity of their own tests.