Temporal validity as meta-science

Abstract

The “credibility revolution” has forced quantitative social scientists to confront the limits of our methods for creating general knowledge. As a result, many practitioners aim to generate valid but local knowledge and then synthesize and apply that knowledge to predict what will happen in a target context. Positivist social science has until recently been hamstrung with other, more immediate threats to validity and inference, but I argue that recent advances in statistical approaches to the problem of external validity reveal limits of the current paradigm. This article and the term “temporal validity” illustrate the intrinsic limits of agnostic (i.e., assumption-free) external validity when the target setting is in the future. These limits, I argue, suggest a re-orientation of social science methodology. We should acknowledge that no research design, no empirical knowledge, is perfectible; instead, we should explicitly aim to increase the amount and quality of knowledge we produce. I believe it is useful to characterize this perspective as part of “Meta-Science,” an emerging paradigm within the social sciences. “Temporal validity” and the implied “knowledge decay” thus represent a meta-scientific intervention aimed at increasing the usefulness of the knowledge we produce. Among other structural reforms, I argue that the binary reality of academic scholarship (a paper is published or not) reifies the perfectibility of empirical knowledge and is thus an impediment to recognizing the continuous nature of all forms of scientific validity.

Can Generative AI improve social science?

“…This may create “drift” within LLMs, wherein improving model performance in one area might change the outputs they produce in another domain ( 74 ). Finally, social scientists must consider broader forms of temporal validity ( 13 , 75 ). As LLMs evolve in response to user behavior in different ways—as well as ongoing events in the world—this will create another significant challenge for those who strive for reproducible research with Generative AI.…”

A critical review of the use of cognitive ability testing for selection into graduate and higher professional occupations

“…Temporal validity (Austin et al., 2016; Munger, 2019) refers to the extent to which research findings remain valid (i.e. generalize) over time.…”

Testing, Testing: Identifying Contemporary Analytics Practices in Digital Politics

“…can and do change over time (Karpf, 2016). Munger (2022) refers to this as "temporal validity," arguing that most academic science is done with the goal of predicting the future; conversely, in digital analytics the goal of an individual A/B test is grounded in making a decision for the present moment, answering the question "what works better, right now?" If list members' material experiences of the Internet change rapidly in terms of device, screen size, or email client, then digital testers should not necessarily expect the findings from an individual test to hold across multiple experiments, if the material and/or temporal conditions themselves are key to the results.…”