Forming conceptually rich social categories helps people navigate the complex social world by allowing them to reason about others’ likely thoughts, beliefs, actions, and interactions as guided by group membership. Yet, social categorization often has nefarious consequences. We suggest that the foundation of the human ability to form useful social categories is in place in infancy: social categories guide infants’ inferences about peoples’ shared characteristics and social relationships. We also suggest that the ability to form abstract social categories may be separable from the eventual negative downstream consequences of social categorization, including prejudice, discrimination and stereotyping. Whereas a tendency to form inductively rich social categories appears early in ontogeny, prejudice based on each particular category dimension may not be inevitable.