Replications, Comparisons, Sampling and the Problem of Representativeness in Animal Cognition Research

Abstract

Research in animal behaviour often involves small and idiosyncratic samples. This can constrain the generalizability and replicability of study’s results and prevent meaningful comparisons between samples. However, there is little consensus about what constitutes a replication and what makes a strong comparison in animal research. We apply a resampling definition of replication to answer these questions in Part 1 of this article, and in Part 2, we focus on the problem of representativeness in animal research. Through three case studies, we highlight how and when representativeness may be an issue in animal behavior and cognition research, and show how the representativeness problems can be viewed through the lenses of, i) replicability, ii) generalizability and external validity, iii) pseudoreplication and, iv) theory testing. Next, we discuss when and how researchers can improve their ability to learn from small sample research through, i) increasing heterogeneity in experimental design, ii) increasing homogeneity in experimental design, and, iii) statistically modeling variation. Finally, we describe how the strongest solutions will vary depending on the goals and resources of individual research programmes and discuss some barriers towards implementing them.

Doing Reliable Research in Comparative Psychology: Challenges and Proposals for Improvement

“…Despite the scarcity of evidence regarding the reliability of comparative psychology findings, researchers in ecology and evolution (fields closely related to ours) report engaging in QRPs at comparable levels to other areas of psychology, with 51% of respondents reporting engaging in HARKing, 42% collecting additional data after conducting statistical analyses to check for significance, and 64% selectively reporting 'positive' results and not including non-significant findings (Fraser et al, 2018). On a more positive note, a recent survey of 63 papers in the animal physical cognition literature showed that some papers (10-17%) did report negative findings, and a further 24-46% made inconclusive claims, demonstrating that 'negative' findings can and have been published (Farrar et al, 2021).…”

“…Even within a single strain of mice housed in tightly controlled, highly standardized lab environments, there is evidence that systematic differences in stress-like behaviours can occur across labs (Crabbe et al, 1999). A re-analysis of data from Mac-Lean et al ( 2014)-a mutli-site, multi-species comparison of performance in two inhibitory control tasksrevealed some instances of large within-species variation in performance between testing sites (Farrar et al, 2021). Squirrel monkeys' performance on the cylinder task was 60% correct in Edinburgh, compared with only 5% correct in Kyoto.…”

“…Squirrel monkeys' performance on the cylinder task was 60% correct in Edinburgh, compared with only 5% correct in Kyoto. The key point here is that had the squirrel monkeys in Kyoto instead been, say, capuchin monkeys, then this difference in performance would likely have been attributed to a species difference in cognitive ability, rather than a difference due to test site (Farrar et al, 2021).…”

“…It is important to bear in mind that replication should be viewed as a process of evidence-building, rather than individual replications being considered as successes or failures (Edlund et al, 2021). There are many reasons that a comparative psychology replication study might not produce the same findings, aside from the possibility that the initial findings were spurious, such as seasonal and developmental variation in behaviour (Farrar et al, 2021). In addition, the earlier section on Comparing across testing sites highlighted numerous variables that can influence animals' performance in experimental tasks, including experimenter, housing arrangements, and prior experience.…”

A Longitudinal Study of Great Ape Cognition: Stability, Reliability and the Influence of Individual Characteristics

“…Psychological science has been riddled with problems of non-replicable results (Collaboration, 2015). Animal cognition research shows many of the characteristics that have been identified to yield a low replication rate in other psychological fields (Farrar et al, 2020b;Stevens, 2017). Furthermore, replication attempts are rare in animal cognition research (Farrar et al, 2020a).…”