Degrees of riskiness, falsifiability, and truthlikeness

Vignero, Leander; Wenmackers, Sylvia

doi:10.1007/s11229-021-03310-5

Cited by 3 publications

(8 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There is a nuanced relationship between the plausible falsifiability measured by KL-delta and the general notion of model complexity. Falsifiability is certainly not the only way to conceive of model complexity, although we agree with Pashler (2000, 2002), Veksler et al (2015), Vanpaemel (2009Vanpaemel ( , 2020, Vignero and Wenmackers (2021), and others that it provides a particularly useful perspective.…”

Section: Intuitions About Complexity and Falsifiabilitysupporting

confidence: 78%

“…As Vanpaemel (2020) and Vignero and Wenmackers (2021) note, putting an emphasis on plausible failure rather than just possible failure corresponds to the philosophical difference between logical falsification (Popper, 1934) and surprising or unexpected evidence (Lakatos, 1978). Considerations of plausibility underlie the notion of critical tests in scientific experimentation and the systematic comparison of competing hypotheses (Platt, 1964).…”

Section: Measuring Plausible Falsifiabilitymentioning

confidence: 99%

“…The entropy of a model's predictions could be called the prior predictive entropy and corresponds to what Vignero and Wenmackers (2021) call "informativeness." Equation 8 shows that under a uniform data prior, one model will be simpler than another if its prior predictive entropy is lower.…”

Section: How Kl-delta Measures Possible and Plausible Falsifiabilitymentioning

confidence: 99%

“…This additional element is needed to distinguish the falsifiability of the nephew's two predictions from the niece's two predictions. It corresponds to what Vignero and Wenmackers (2021) call "conceptual novelty," capturing the extent to which "a hypothesis may predict phenomena that have no This document is copyrighted by the American Psychological Association or one of its allied publishers.…”

Section: How Kl-delta Measures Possible and Plausible Falsifiabilitymentioning

confidence: 99%

“…The relative precision of your niece’s and nephew’s predictions means there are more outcomes that would not match their predictions. The second idea is the role of plausible falsifiability in making models more or less complex (Roberts & Pashler, 2000, 2002; Vanpaemel, 2020; Vignero & Wenmackers, 2021). It is more plausible that your nephew’s predictions will be falsified when compared to your niece’s predictions based on what is known about likely outcomes.…”

mentioning

confidence: 99%

See 4 more Smart Citations

Evaluating the complexity and falsifiability of psychological models.

Villarreal¹,

Etz²,

Lee³

2023

Psychological Review

View full text Add to dashboard Cite

Understanding model complexity is important for developing useful psychological models. One way to think about model complexity is in terms of the predictions a model makes and the ability of empirical evidence to falsify those predictions. We argue that existing measures of falsifiability have important limitations and develop a new measure. KL-delta uses Kullback–Leibler divergence to compare the prior predictive distributions of models to the data prior that formalizes knowledge about the plausibility of different experimental outcomes. Using introductory conceptual examples and applications with existing models and experiments, we show that KL-delta challenges widely held scientific intuitions about model complexity and falsifiability. In a psychophysics application, we show that hierarchical models with more parameters are often more falsifiable than the original nonhierarchical model. This counters the intuition that adding parameters always makes a model more complex. In a decision-making application, we show that a choice model incorporating response determinism can be harder to falsify than its special case of probability matching. This counters the intuition that if one model is a special case of another, the special case must be less complex. In a memory recall application, we show that using informative data priors based on the serial position curve allows KL-delta to distinguish models that otherwise would be indistinguishable. This shows the value in model evaluation of extending the notion of possible falsifiability, in which all data are considered equally likely, to the more general notion of plausible falsifiability, in which some data are more likely than others.

show abstract

Section: Intuitions About Complexity and Falsifiabilitysupporting

confidence: 78%

Section: Measuring Plausible Falsifiabilitymentioning

confidence: 99%

Section: How Kl-delta Measures Possible and Plausible Falsifiabilitymentioning

confidence: 99%

Section: How Kl-delta Measures Possible and Plausible Falsifiabilitymentioning

confidence: 99%

mentioning

confidence: 99%

See 3 more Smart Citations

Evaluating the complexity and falsifiability of psychological models.

Villarreal¹,

Etz²,

Lee³

2023

Psychological Review

View full text Add to dashboard Cite

show abstract

Causal Factor Investing

López de Prado

2023

View full text Add to dashboard Cite

Virtually all journal articles in the factor investing literature make associational claims, instead of causal claims. Authors do not identify the causal graph consistent with the observed phenomenon, they justify their chosen model specification in terms of correlations, and they do not propose experiments for falsifying causal mechanisms. Absent a causal theory, their findings are likely false, due to rampant backtest overfitting and incorrect specification choices. This Element differentiates between type-A and type-B spurious claims, and explains how both types prevent factor investing from advancing beyond its current phenomenological stage. It analyzes the current state of causal confusion in the factor investing literature, and proposes solutions with the potential to transform factor investing into a truly scientific discipline. This title is also available as Open Access on Cambridge Core.

show abstract

The Hierarchy of Empirical Evidence in Finance

Prado

2023

SSRN Journal

View full text Add to dashboard Cite

Degrees of riskiness, falsifiability, and truthlikeness

Cited by 3 publications

References 29 publications

Evaluating the complexity and falsifiability of psychological models.

Evaluating the complexity and falsifiability of psychological models.

Causal Factor Investing

The Hierarchy of Empirical Evidence in Finance

Contact Info

Product

Resources

About