Putting psychology to the test: Rethinking model evaluation through benchmarking and prediction

Abstract

Consensus on standards for evaluating models and theories is an integral part of every science. Nonetheless, in psychology, relatively little focus has been placed on defining reliable communal metrics to assess model performance. Evaluation practices are often idiosyncratic, and are affected by a number of shortcomings (e.g., failure to assess models' ability to generalize to unseen data) that make it difficult to discriminate between good and bad models. Drawing inspiration from fields like machine learning and statistical genetics, we argue in favor of introducing common benchmarks as a means of overcoming the lack of reliable model evaluation criteria currently observed in psychology. We discuss a number of principles benchmarks should satisfy to achieve maximal utility; identify concrete steps the community could take to promote the development of such benchmarks; and address a number of potential pitfalls and concerns that may arise in the course of implementation. We argue that reaching consensus on common evaluation benchmarks will foster cumulative progress in psychology, and encourage researchers to place heavier emphasis on the practical utility of scientific models.

Age-related nuances in knowledge assessment

“…Instead a broad (best representative) sampling from the item universe is necessary to abstain from trimming and artificially homogenizing item samples which would result in oversimplified models. These two views in dealing with item uniqueness are also addressed by Rocca and Yarkoni (2020), who distinguished between a model evaluation that follows an explanatory tradition and those that follows a prediction-oriented approach. That is, instead of developing models that try to explain the underlying processes (i.e., data modeling), researchers could also evaluate models based on their ability to predict a specific outcome (i.e., algorithmic modeling), regardless of how this aim is achieved.…”

The Dual Mechanisms of Cognitive Control dataset, a theoretically-guided within-subject task fMRI battery

“…Of the 55 participants included in DMCC55B, 31 were previously participants in the HCP Young Adult study (Van Essen et al 2012), enabling researchers with HCP data access to combine data from the two studies. The DMCC55B can also be used as a benchmark dataset (Varoquaux 2018;Willems, Nastase, and Milivojevic 2020;Rocca and Yarkoni 2020), particularly for methodological investigations focused on higher cognitive brain functions or regions.…”

Validation and Generalizability of Machine Learning Prediction Models on Attrition in Longitudinal Studies

“…Another common drawback of traditional attrition modeling approaches is that it is unclear whether theirs results are generalizable. The ability of a model to provide accurate and generalizable predictions is especially essential in applied research (Rocca & Yarkoni, 2020;Shmueli, 2010) such as study retention. To enable panel administrators to employ effective retention strategies (e.g., person-specific incentives at future waves), a prediction model also has to hold in future waves.…”