A prominent vein of comparative cognition research asks which cognitive abilities may be ascribed to different species. Here, we argue that the current structure of comparative cognition makes it near impossible to evaluate the accuracy of many of the claims produced by the field's empirical research. We base this argument on six observations: i) the field is biased towards confirming more exceptional abilities in animals, ii) a likely high rate of false positive discovery, iii) the persistence of the bias towards more exceptional abilities, even in the presence of strong methodological criticism, iv) the absence of a formal method to assess evidence of absence of a cognitive ability, v) the ambiguity in definitions used to make claims, and vi) the small size of the research field. We then highlight how the widespread methodological and conceptual criticism in the comparative cognition literature does not facilitate scientific progress, rather it can impede it by creating an illusion of scientific rigour. Finally, we call for a reform of our research field that draws on recent developments in related fields and the Open Science movement. Now, fifteen months later, we have finally been able to put these thoughts to paper -the result of which is in front of you. When we are engrossed in a research programme, most of our time and effort is taken up by it. Taking a step back to evaluate the framework itself can be difficult. We hope that this paper may aid others in doing so if and when they decide to take that step.