A Model Hierarchy for Psychological Science

Kellen, David

doi:10.1007/s42113-019-00037-y

Cited by 55 publications

(41 citation statements)

References 46 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Somewhat surprisingly, since starting to work on this paper in early 2019, numerous contributions in different areas of psychology have identified this crisis of theory as a crucial challenge moving forward (Borsboom et al, 2020;Burger et al, 2019;Gershman, 2019;Guest & Martin, 2020;Haslbeck et al, 2019;Kellen, 2019;Lakens & Debruine, 2020;Muthukrishna & Henrich, 2019;Oberauer & Lewandowsky, 2019;Robinaugh, Haslbeck, et al, 2019;Savi et al, 2019;Smaldino, 2019;Szollosi et al, 2019;Van Rooij & Baggio, 2020), and both the Journal of Abnormal Psychology and Perspective of Psychological Science have opened calls for special issues on theory.…”

Section: Putting the Theory Back Into Psychologymentioning

confidence: 99%

Lack of theory building and testing impedes progress in the factor and network literature

Fried¹

2020

Preprint

113

138

View full text Add to dashboard Cite

The last decade has brought reforms to improve methodological practices, with the goal to increase the reliability and replicability of effects. However, explanations of effects remain scarce, and a growing chorus of scholars argues that the replicability crisis has distracted from a crisis of theory. In the same decade, the empirical literature using factor and network models has grown rapidly. I discuss three ways in which this literature falls short of theory building and testing. First, statistical and theoretical models are conflated, leading to invalid inferences such as the existence of psychological constructs based on factor models, or recommendations for clinical interventions based on network models. I demonstrate this inferential gap in a simulation study on statistical equivalence: excellent model fit does little to corroborate a theory, regardless of quality or quantity of data. Second, researchers fail to explicate theories about psychological constructs, but use implicit causal beliefs to guide inferences. These latent theories have led to problematic best practices in psychological research where inferences are drawn based on one specific causal model that is assumed, but not explicated. Third, explicated theories are often weak theories: narrative and imprecise descriptions vulnerable to hidden assumptions and unknowns. They fail to make clear predictions, and it remains unclear whether statistical effects corroborate such theories or not. Weak theories are immune to refutation or revision. I argue that these three challenges to theory building and testing are common and harmful, and impede theory formation, failure, and reform. A renewed focus on theoretical psychology and formal models offers a way forward.

show abstract

Section: Putting the Theory Back Into Psychologymentioning

confidence: 99%

Lack of theory building and testing impedes progress in the factor and network literature

Fried¹

2020

Preprint

113

138

View full text Add to dashboard Cite

show abstract

“…Explanation requires (a) a theory encoding core causal assumptions about the phenomenon of interest, (b) experimental tasks or data sources that capture the key theoretical phenomenon, and often (c) statistical models that test theoretical principles against observed data while accounting for uncertainty (Guest & Martin, 2020;Kellen, 2019;Suppes, 1966). In the social, behavioral, and brain sciences, theories may be specified verbally, conceptually, or (less commonly) mathematically.…”

Section: Introductionmentioning

confidence: 99%

“…Fortunately, such theory-description gaps can be addressed by explicating assumptions of both the descriptive and theoretical models, and by iteratively refining them in a mutually constraining fashion (Guest & Martin, 2020;Kellen, 2019;Suppes, 1966).…”

Section: Introductionmentioning

confidence: 99%

Theoretically Informed Generative Models Can Advance the Psychological and Brain Sciences: Lessons from the Reliability Paradox

Haines¹,

Pd²,

Lh³

et al. 2020

Preprint

106

119

View full text Add to dashboard Cite

Behavioral tasks (e.g., Stroop task) that produce replicable group-level effects (e.g., Stroop effect) often fail to reliably capture individual differences between participants (e.g., low test-retest reliability). This “reliability paradox” has led many researchers to conclude that most behavioral tasks cannot be used to develop and advance theories of individual differences. However, these conclusions are derived from statistical models that provide only superficial summary descriptions of behavioral data, thereby ignoring theoretically-relevant data-generating mechanisms that underly individual-level behavior. More generally, such descriptive methods lack the flexibility to test and develop increasingly complex theories of individual differences. To resolve this theory-description gap, we present generative modeling approaches, which involve using background knowledge to specify how behavior is generated at the individual level, and in turn how the distributions of individual-level mechanisms are characterized at the group level—all in a single joint model. Generative modeling shifts our focus away from estimating descriptive statistical “effects” toward estimating psychologically meaningful parameters, while simultaneously accounting for measurement error that would otherwise attenuate individual difference correlations. Using simulations and empirical data from the Implicit Association Test and Stroop, Flanker, Posner Cueing, and Delay Discounting tasks, we demonstrate how generative models yield (1) higher test-retest reliability estimates, and (2) more theoretically informative parameter estimates relative to traditional statistical approaches. Our results reclaim optimism regarding the utility of behavioral paradigms for testing and advancing theories of individual differences, and emphasize the importance of formally specifying and checking model assumptions to reduce theory-description gaps and facilitate principled theory development.

show abstract

“…Also, one should keep in mind that these data are an amalgamation of different strategies and beliefs that are unlikely to be successfully captured by any single model. In this situation it is best to focus on the qualitative aspects of models (for discussions, see Kellen, 2019;Navarro, 2019).…”

Section: Interim Discussionmentioning

confidence: 99%

“…The minimal requirement for any candidate model is that it succeeds in capturing the data at hand (Bogen & Woodward, 1988). Given that we are dealing with aggregate data, it is reasonable to focus our evaluation on the models' ability to capture the qualitative patterns found in the data (for discussions, see Kellen, 2019;Navarro, 2019;Shiffrin & Nobel, 1997). It turns out that the Rule of Succession (as well as its generalization) cannot fulfill this requirement.…”

Section: Addressing Concerns On Model Flexibility Evaluation and Comentioning

confidence: 99%

Normative Accounts of Illusory Correlations

Bott¹,

Kellen²,

Klauer³

2019

Preprint

Self Cite

View full text Add to dashboard Cite

When learning about the joint occurrence of different variables, individuals often manifest biases in the associations they infer. In some cases they infer an association when none is present in the observed sample. Other times they infer an association that is contrary to the one that is in fact observed. These illusory correlations are often interpreted as being the byproduct of selective processing or as the outcome of an ‘illogical’ pseudocontingency heuristic. More recently, a normative account of illusory correlations has been proposed, according to which they result from an application of Laplace’s Rule of Succession. The present work will discuss the empirical and theoretical limitations associated with this normative account, and argue for its dismissal. As an alternative, we propose a normative account that casts illusory correlations as the expected outcome of a Bayesian reasoner relying on marginal frequencies. We show that this novel account succeeds in capturing the qualitative patterns found in a corpus of published studies.

show abstract

A Model Hierarchy for Psychological Science

Cited by 55 publications

References 46 publications

Lack of theory building and testing impedes progress in the factor and network literature

Lack of theory building and testing impedes progress in the factor and network literature

Theoretically Informed Generative Models Can Advance the Psychological and Brain Sciences: Lessons from the Reliability Paradox

Normative Accounts of Illusory Correlations

Contact Info

Product

Resources

About