Social complexity has been one of the recent emerging topics in the study of animal and human societies, but the concept remains both poorly defined and understood. In this paper, I critically review definitions and studies of social complexity in invertebrate and vertebrate societies, arguing that the concept is being used inconsistently in studies of vertebrate sociality. Group size and cohesion define one cornerstone of social complexity, but the nature and patterning of social interactions contribute more to interspecific variation in social complexity in species with individual recognition and repeated interactions. Humans provide the only example where many other unique criteria are used, and they are the only species for which intraspecific variation in social complexity has been studied in detail. While there is agreement that complex patterns emerge at the group level as a result of simple interactions and as a result of cognitive abilities, there is consensus neither on their relative importance nor on the role of specific cognitive abilities in different lineages. Moreover, aspects of reproduction and parental care have also been invoked to characterize levels of social complexity, so that no single comprehensive measure is readily available. Because even fundamental components of social complexity are difficult to compare across studies and species because of inconsistent definitions and operationalization of key social traits, I define and characterize social organization, social structure, mating system, and care system as distinct components of a social system. Based on this framework, I outline how different aspects of the evolution of social complexity are being studied and suggest questions for future research. Significance statement Animal and human societies differ in social complexity, i.e., the number and association patterns of group members as well as the nature and patterning of their social relationships, but the dimensions of social complexity, the processes that generate it, the selective forces that engender different levels of social complexity, and the evolutionary consequences of this variation remain to be comprehensively understood. Here, I offer a conceptual framework for the systematic and comparative studies of social complexity by defining its main components as well as their proximate and ultimate relationships.