Between the Devil and the Deep Blue Sea: Tensions Between Scientific Judgement and Statistical Model Selection

Navarro, Daniel J.

doi:10.1007/s42113-018-0019-z

Cited by 105 publications

(117 citation statements)

References 37 publications

Supporting

Mentioning

117

Contrasting

Order By: Relevance

“…Given these mathematical realities, the role of theory in psychology can be clarified. Many recent calls for theory in psychology (Muthukrishna & Henrich, 2019;Navarro, 2018;Smaldino, 2019;Stewart…”

Section: Replication Vs Discoverymentioning

confidence: 99%

Practical Methodological Reform Needs Good Theory

Gervais¹

2020

Preprint

View full text Add to dashboard Cite

In the face of unreplicable results, statistical anomalies, and outright fraud, introspection and changes in the psychological sciences have taken root. Vibrant reform and meta-science movements have emerged. These are exciting developments, and may point towards practical improvements in the future. Yet there is nothing so practical as good theory. This paper outlines aspects of reform and meta-science that appear ripe for an injection of theory, including a lot of excellent and overlooked theoretical work from different disciplines. I focus on theories of meta-science and scientific forensics, theories of scientific aspiration, and theories of cultural change in science. There is much room for theoretical growth -- a growth that can only benefit emerging movements -- and highlight some, focusing especially on the potential of cultural evolutionary theories to help direct reform efforts. We have unwittingly evolved a toxic scientific ecosystem; theory may help us intelligently design a better one.

show abstract

Section: Replication Vs Discoverymentioning

confidence: 99%

Practical Methodological Reform Needs Good Theory

Gervais¹

2020

Preprint

View full text Add to dashboard Cite

show abstract

mentioning

confidence: 99%

“…A recent article by Navarro (2018) provided a detailed discussion on both specific and broad issues within the area of model selection. Although this article was a comment on the specific critique of Bayesian leave-one-out cross-validation by Gronau and Wagenmakers (2018a), Navarro (2018) also made several broader points on the philosophy of modelling, and how we should evaluate these formalized theories. These broader points made by Navarro (2018) appear to have been the most impactful part of the entire debate so far, with Navarro's article currently (as of the 18 th of January, 2019) having over 3,400 downloads and 122 shares, compared to the 678 downloads and 11 shares of the original article by Gronau and Wagenmakers.…”

mentioning

confidence: 99%

“…Although this article was a comment on the specific critique of Bayesian leave-one-out cross-validation by Gronau and Wagenmakers (2018a), Navarro (2018) also made several broader points on the philosophy of modelling, and how we should evaluate these formalized theories. These broader points made by Navarro (2018) appear to have been the most impactful part of the entire debate so far, with Navarro's article currently (as of the 18 th of January, 2019) having over 3,400 downloads and 122 shares, compared to the 678 downloads and 11 shares of the original article by Gronau and Wagenmakers. In general, Navarro (2018) suggested that 1) "one of the most important functions of a scientific theory is ... to encourage directed exploration of new territory" (p.3), 2) "understanding how the qualitative patterns in the empirical data emerge naturally from a computational model of a psychological process is often more scientifically useful than presenting a quantified measure of its performance" (p.6), and 3) "quantitative measures of performance are essentially selecting models based on their ancillary assumptions" (p.6).…”

mentioning

confidence: 99%

“…Although Gronau and Wagenmakers (2018b) provided a reply to all three commentaries made on their original article (Vehtari et al, 2018;Navarro, 2018;Chandramouli & Shiffrin, 2018), their response mostly focused on replying to Vehtari et al (2018) with further limitations of Bayesian leave-one-out cross-validation. Their section replying to Navarro (2018) briefly mentioned that quantitative methods are useful as "the data may not yield a clear result at first sight" (p.8), but focused on a more specific point, which was regarding how useful simple examples (or in the more critical terms of Navarro, "toy examples") are in assessing the robustness of analysis methods.…”

mentioning

confidence: 99%

See 2 more Smart Citations

What factors are most important in finding the best model of a psychological process? Comment on Navarro (2018)

Nj¹

2019

Preprint

View full text Add to dashboard Cite

Psychology research has become increasingly focused on creating formalized models of psychological processes, which can make exact quantitative predictions about observed data that are the result of some unknown psychological process, allowing a better understanding of how psychological processes may actually operate. However, using models to understand psychological processes comes with an additional challenge: how do we select the best model from a range of potential models that all aim to explain the same psychological process? A recent article by Navarro (2018; Computational Brain & Behavior ) provided a detailed discussion on several broad issues within the area of model selection,with Navarro suggesting that "one of the most important functions of a scientific theory is ... to encourage directed exploration of new territory" (p.3), that "understanding how the qualitative patterns in the empirical data emerge naturally from a computational model of a psychological process is often more scientifically useful than presenting a quantified measure of its performance" (p.6), and that "quantitative measures of performance are essentially selecting models based on their ancillary assumptions" (p.6). Here, I provide a critique of several of Navarro's points on these broad issues. In contrast to Navarro, I argue that all possible data should be considered when evaluating a process model (i.e., not just data from novel contexts), that quantitative model selection methods provide a more principled and complete method of selecting between process models than visual assessments of qualitative trends, and that the idea of ancillary assumptions that are not part of the core explanation in the model is a slippery slope to an infinitely flexible model.

show abstract

Formal models for the study of the relationship between fluctuating asymmetry and fitness in humans

Farrera

2022

American Journal of Biological Anthropology

View full text Add to dashboard Cite

Objectives To evaluate three of the main verbal models that have been proposed to explain the relationship between fluctuating asymmetry and fitness in humans: the “good genes,” the “good development,” and the “growth” hypotheses. Materials and Methods A formal model was generated for each verbal model following three steps. First, based on the literature, a theoretical causal model and the theoretical object of inquiry were outlined. Second, an empirical causal model and the targets of inference were defined using observational data of facial asymmetries and life‐history traits related to fitness. Third, generalized linear models and causal inference were used as the estimation strategy. Results The results suggest that the theoretical and empirical assumptions of the “good genes” hypothesis should be reformulated. The results were compatible with most of the empirical assumptions of “the good development” hypothesis but suggest that further discussion of its theoretical assumptions is needed. The results were less informative about the “growth” hypothesis, both theoretically and empirically. There was a positive association between facial fluctuating asymmetry and the number of offspring that was not compatible with any of the empirical causal models evaluated. Conclusions Although the three hypotheses focus on different aspects of the link between asymmetry and fitness, their overlap opens the possibility of a unified theory on the subject. The results of this study make explicit which assumptions need to be updated and discussed, facilitating the advancement of this area of research. Overall, this study elucidates the potential benefit of using formal models for theory revision and development.

show abstract

Between the Devil and the Deep Blue Sea: Tensions Between Scientific Judgement and Statistical Model Selection

Cited by 105 publications

References 37 publications

Practical Methodological Reform Needs Good Theory

Practical Methodological Reform Needs Good Theory

What factors are most important in finding the best model of a psychological process? Comment on Navarro (2018)

Formal models for the study of the relationship between fluctuating asymmetry and fitness in humans

Contact Info

Product

Resources

About