Explaining how and why some species evolved to have more complex social structures than others has been a long-term goal for many researchers in animal behavior because it would provide important insight into the links between evolution and ecology, sociality, and cognition. However, despite long-standing interest, the evolution of social complexity is still poorly understood. This may be due in part to researchers focusing on the feasibility of quantifying aspects of sociality, rather than what features are characteristic of animal social complexity in the first place. Any given approach to studying complexity can tell us some things about animal sociality, but may miss others, so it is critical to decide first how to conceptualize complexity before jumping in to quantifying it. Here, we briefly summarize five existing approaches to measuring social complexity. Then, we highlight three fundamental concepts that are commonly used in the field of complex systems: (1) scales of organization, (2) compression, and (3) emergence. All of these concepts are applicable to the study of animal social systems, but are not often explicitly addressed in existing social complexity measures. We discuss how these concepts can provide a rigorous foundation for conceptualizing social complexity, the potential benefits of incorporating them, and how existing measures do (or do not) include them. Ultimately, researchers need to critically evaluate any measure of animal social complexity in order to balance the biological relevance of the aspect of sociality they are quantifying with the feasibility of obtaining enough data.