The interplay of natural selection and genetic drift, influenced by geographic isolation, mating systems and population size, determines patterns of genetic diversity within species. The sperm whale provides an interesting example of a long-lived species with few geographic barriers to dispersal. Worldwide mtDNA diversity is relatively low, but highly structured among geographic regions and social groups, attributed to female philopatry. However, it is unclear whether this female philopatry is due to geographic regions or social groups, or how this might vary on a worldwide scale. To answer these questions, we combined mtDNA information for 1091 previously published samples with 542 newly obtained DNA profiles (394-bp mtDNA, sex, 13 microsatellites) including the previously unsampled Indian Ocean, and social group information for 541 individuals. We found low mtDNA diversity (π = 0.430%) reflecting an expansion event <80 000 years bp, but strong differentiation by ocean, among regions within some oceans, and among social groups. In comparison, microsatellite differentiation was low at all levels, presumably due to male-mediated gene flow. A hierarchical amova showed that regions were important for explaining mtDNA variance in the Indian Ocean, but not Pacific, with social group sampling in the Atlantic too limited to include in analyses. Social groups were important in partitioning mtDNA and microsatellite variance within both oceans. Therefore, both geographic philopatry and social philopatry influence genetic structure in the sperm whale, but their relative importance differs by sex and ocean, reflecting breeding behaviour, geographic features and perhaps a more recent origin of sperm whales in the Pacific. By investigating the interplay of evolutionary forces operating at different temporal and geographic scales, we show that sperm whales are perhaps a unique example of a worldwide population expansion followed by rapid assortment due to female social organization.