Referential gestures are of pivotal importance to the human species. We effortlessly make use of each others' referential gestures to attend to the same things, and our ability to use these gestures show themselves from very early in life. Almost 20 years ago, James Anderson and colleagues presented an experimental paradigm with which to examine the use of referential gestures in non-human primates: the object-choice task. Since then, numerous object-choice studies have been made, not only with primates but also with a range of other animal taxa. Surprisingly, several non-primate species appear to perform better in the object-choice task than primates do. Different hypotheses have been offered to explain the results. Some of these have employed generalizations about primates or subsets of primate taxa that do not take into account the unparalleled diversity that exists between species within the primate order on parameters relevant to the requirements of the object-choice task, such as social structure, feeding ecology, and general morphology. To examine whether these broad primate generalizations offer a fruitful organizing framework within which to interpret the results, a review was made of all published primate results on the use of gazing and glancing cues with species ordered along the primate phylogenetic tree. It was concluded that differences between species may be larger than differences between ancestry taxa, and it is suggested that we need to start rethinking why we are testing animals on experimental paradigms that do not take into account what are the challenges of their natural habitat.