The question as to why primates have evolved unusually large brains has received much attention, with many alternative proposals all supported by evidence. We review the main hypotheses, the assumptions they make and the evidence for and against them. Taking as our starting point the fact that every hypothesis has sound empirical evidence to support it, we argue that the hypotheses are best interpreted in terms of a framework of evolutionary causes (selection factors), consequences (evolutionary windows of opportunity) and constraints (usually physiological limitations requiring resolution if large brains are to evolve). Explanations for brain evolution in birds and mammals generally, and primates in particular, have to be seen against the backdrop of the challenges involved with the evolution of coordinated, cohesive, bonded social groups that require novel social behaviours for their resolution, together with the specialized cognition and neural substrates that underpin this. A crucial, but frequently overlooked, issue is that fact that the evolution of large brains required energetic, physiological and time budget constraints to be overcome. In some cases, this was reflected in the evolution of ‘smart foraging’ and technical intelligence, but in many cases required the evolution of behavioural competences (such as coalition formation) that required novel cognitive skills. These may all have been supported by a domain-general form of cognition that can be used in many different contexts.This article is part of the themed issue ‘Physiological determinants of social behaviour in animals’.