Psychologists psychologizing scientific psychology: An epistemological reading of the replication crisis

Flis, Ivan

doi:10.1177/0959354319835322

Cited by 52 publications

(71 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…How should we assess the effectiveness of the proposed methodological solutions? In the absence of a clear philosophy of science (cf., Flis, 2019), it becomes difficult to provide arguments for why direct replication or preregistration are important, because we do not know whether what they aim to achieve coincides with the goals of science. Equally difficult is to know what to do to improve theory development.…”

Section: Introductionmentioning

confidence: 99%

Arrested theory development: The misguided distinction between exploratory and confirmatory research

Szöllősi¹,

Donkin²

2019

Preprint

View full text Add to dashboard Cite

Starting from the view that progress in science consists of the improvement of our theories, in the current paper we ask two questions: what makes a theory good, and how much do the current method-oriented solutions to the replication crisis contribute to the development of good theories? Based on contemporary philosophy of science, we argue that good theories are hard-to-vary: they (1) explain what they are supposed to explain, (2) are consistent with other good theories, and (3) cannot easily be adapted to explain anything. Theories can be improved by identifying problems in them either by argument or by experimental test, and then correcting these problems by changing the theory. Importantly, such changes and the resultant theory should only be assessed based on whether they are hard-to-vary. An assessment of the current state of the behavioral sciences reveals that theory development is arrested by the lack of consideration for how easy it is to change theories to account for unexpected observations. Further, most of the current method-oriented solutions are unlikely to contribute much to the development of good theories, because they do not work towards eliminating this problem. Instead, they reward only temporary inflexibility in theories, and promote the assessment of theory change based on whether the theory was changed before (confirmatory) or after (exploratory) an experimental test, but not whether that change yields a hard-to-vary theory. Finally, we argue that these methodological solutions would become irrelevant if we turned our focus to the explicit aim of developing theories that are hard-to-vary.

show abstract

Section: Introductionmentioning

confidence: 99%

Arrested theory development: The misguided distinction between exploratory and confirmatory research

Szöllősi¹,

Donkin²

2019

Preprint

View full text Add to dashboard Cite

show abstract

“…In the present crisis in psychology, the field as a whole has in fact engaged in reflexivity in this sense. The psychology of psychologists, with their biases and motivated reasoning and other forms of cognitive flaws, has been much-discussed (Flis 2019;Morawski 2020). Psychologists have shown themselves to be suspicious of their own minds and skeptical of their ability to be rational, unless disciplined and constrained by procedures such as pregistration.…”

mentioning

confidence: 99%

Experimenter as automaton; experimenter as human: exploring the position of the researcher in scientific research

Field

Derksen

2020

Euro Jnl Phil Sci

View full text Add to dashboard Cite

The crisis of confidence in the social sciences has many corollaries which impact our research practices. One of these is a push towards maximal and mechanical objectivity in quantitative research. This stance is reinforced by major journals and academic institutions that subtly yet certainly link objectivity with integrity and rigor. The converse implication of this may be an association between subjectivity and low quality. Subjectivity is one of qualitative methodology’s best assets, however. In qualitative methodology, that subjectivity is often given voice through reflexivity. It is used to better understand our own role within the research process, and is a means through which the researcher may oversee how they influence their research. Given that the actions of researchers have led to the poor reproducibility characterising the crisis of confidence, it is worthwhile to consider whether reflexivity can help improve the validity of research findings in quantitative psychology. In this report, we describe a combination approach of research: the data of a series of interviews helps us elucidate the link between reflexive practice and quality of research, through the eyes of practicing academics. Through our exploration of the position of the researcher in their research, we shed light on how the reflections of the researcher can impact the quality of their research findings, in the context of the current crisis of confidence. The validity of these findings is tempered, however, by limitations to the sample, and we advise caution on the part of our audience in their reading of our conclusions.

show abstract

“…KEYWORDS: Physical cognition, folk physics, evidence, statistical inference, publication bias Povinelli's (2000) "Folk Physics for Apes" has had a lasting impact on the study of what animals understand about the physical world, their "folk physics". However, in "Folk Physics for Apes", Povinelli also highlighted many of the issues that have been foregrounded in science's current replication crisis (see Flis, 2019), such as replication (Open Science Collaboration, 2015;Povinelli, 2003, p. 16;Zwaan, Etz, et al, 2018) falsification (LeBel et al, 2017), strong inference (Chamberlin, 1965;Open Science Collaboration, 2015;Platt, 1964;Stevens, 2017) and being willing and able to interpret negative results (Fanelli, 2012;Daniël Lakens, 2017;Povinelli, 2003, p. 50;Rosenthal, 1979). In the current paper, we use data from published experiments in animal physical cognition in order to explore how the field has used statistical inferences to make claims, in light of the current replicability issues science faces.…”

mentioning

confidence: 99%

Claims and statistical inference in animal physical cognition research.

Bg¹,

Fischer²,

J³

et al. 2020

Preprint

View full text Add to dashboard Cite

Twenty years after Povinelli’s “Folk Physics for Apes”, this paper assesses how researchers have made claims about animal physical cognition, and the statistical inferences that have been used to support them. These data are relevant in light of the current replicability issues facing science. We surveyed 116 published experiments from 63 papers on physical cognition, which included data from 43 different species of animals. Across these experiments most sample sizes were small, with often fewer than 10 animals being tested. However, in contrast to related psychological disciplines, we found that only 62% of our sample of physical cognition research made positive claims. This suggests that animal physical cognition does not have a strong publication bias towards positive results. Furthermore, we found evidence that researchers are making many true statistical inferences at the individual level, i.e. whether individual animals pass certain tests of physical cognition or not. In contrast, the strength of evidence of statistical effects at the group level was weaker and consistent with many effect sizes being overestimated. Overall, our analysis provides a cautiously optimistic analysis of reliability and bias in animal physical cognition research, however it is nevertheless likely that a non-negligible proportion of results will be difficult to replicate.

show abstract

Psychologists psychologizing scientific psychology: An epistemological reading of the replication crisis

Cited by 52 publications

References 43 publications

Arrested theory development: The misguided distinction between exploratory and confirmatory research

Arrested theory development: The misguided distinction between exploratory and confirmatory research

Experimenter as automaton; experimenter as human: exploring the position of the researcher in scientific research

Claims and statistical inference in animal physical cognition research.

Contact Info

Product

Resources

About