A prominent vein of comparative cognition research asks which cognitive abilities may be ascribed to different species. Here, we argue that the current structure of comparative cognition makes it near impossible to evaluate the accuracy of many of the claims produced by the field's empirical research. We base this argument on six observations: i) the field is biased towards confirming more exceptional abilities in animals, ii) a likely high rate of false positive discovery, iii) the persistence of the bias towards more exceptional abilities, even in the presence of strong methodological criticism, iv) the absence of a formal method to assess evidence of absence of a cognitive ability, v) the ambiguity in definitions used to make claims, and vi) the small size of the research field. We then highlight how the widespread methodological and conceptual criticism in the comparative cognition literature does not facilitate scientific progress, rather it can impede it by creating an illusion of scientific rigour. Finally, we call for a reform of our research field that draws on recent developments in related fields and the Open Science movement. Now, fifteen months later, we have finally been able to put these thoughts to paper -the result of which is in front of you. When we are engrossed in a research programme, most of our time and effort is taken up by it. Taking a step back to evaluate the framework itself can be difficult. We hope that this paper may aid others in doing so if and when they decide to take that step.

Corvids appear to be capable of adjusting their behaviour according to another's perspective, knowledge and desire. For example, Eurasian jays have been found to employ a variety of cache protection strategies to minimise cache loss by responding to cues about the visual perspective or current desire of an observing conspecific. However, it is not known whether these jays (or any other corvid) can integrate multiple cues about different mental states and perform the optimal response accordingly. Across five experiments, we found little evidence that our Eurasian jays responded to either the visual perspective or current desire of another agent. In Experiments 1 and 2 we investigated whether Eurasian jays can limit the risk of cache loss by responding simultaneously to cues about the desire and perspective of a potential conspecific pilferer. Building on established paradigms, we used opaque and clear barriers to manipulate the observer's visual access to cache locations, and specific satiety to manipulate the observer's desire towards different types of food. Across both experiments the jays' caching pattern provided no evidence that they could integrate information about the observer's desire and perspective. Moreover, the results were also inconsistent with the previously reported effects that jays protect their caches by responding to either the visual access or specific satiety of the observer independently. To gain further insight into these unexpected results, we conducted three more experiments. In Experiments 3 and 4, we attempted to replicate the previous finding that Eurasian jays prefer to cache behind an opaque barrier over a clear barrier when observed by a conspecific than when caching in private. In Experiment 5, we further investigated the previous finding that jays preferentially cache a type of food that had been eaten to satiety by a conspecific over a food that had not been eaten by the conspecific. Experiments 3, 4 and 5 found no significant effects in the direction of the previously reported effects, questioning their robustness. We conclude by discussing the implications of our study for the field of corvid cognition and highlight several key issues that affect the reliability of comparative cognition research.

Twenty years after Povinelli’s “Folk Physics for Apes”, this paper assesses how researchers have made claims about animal physical cognition, and the statistical inferences that have been used to support them. These data are relevant in light of the current replicability issues facing science. We surveyed 116 published experiments from 63 papers on physical cognition, which included data from 43 different species of animals. Across these experiments most sample sizes were small, with often fewer than 10 animals being tested. However, in contrast to related psychological disciplines, we found that only 62% of our sample of physical cognition research made positive claims. This suggests that animal physical cognition does not have a strong publication bias towards positive results. Furthermore, we found evidence that researchers are making many true statistical inferences at the individual level, i.e. whether individual animals pass certain tests of physical cognition or not. In contrast, the strength of evidence of statistical effects at the group level was weaker and consistent with many effect sizes being overestimated. Overall, our analysis provides a cautiously optimistic analysis of reliability and bias in animal physical cognition research, however it is nevertheless likely that a non-negligible proportion of results will be difficult to replicate.