Conflict with conspecifics from neighbouring groups over territory, mating opportunities and other resources is observed in many social organisms, including humans. Here we investigate the evolutionary origins of social instincts, as shaped by selection resulting from between-group conflict in the presence of a collective action problem. We focus on the effects of the differences between individuals on the evolutionary dynamics. Our theoretical models predict that high-rank individuals, who are able to usurp a disproportional share of resources in within-group interactions, will act seemingly altruistically in between-group conflict, expending more effort and often having lower reproductive success than their low-rank group-mates. Similar behaviour is expected for individuals with higher motivation, higher strengths or lower costs, or for individuals in a leadership position. Our theory also provides an evolutionary foundation for classical equity theory, and it has implications for the origin of coercive leadership and for reproductive skew theory.

The majority of human societies allow polygynous marriage, and the prevalence of this practice is readily understood in evolutionary terms. Why some societies prescribe monogamous marriage is however not clear: current evolutionary explanations—that social monogamy increases within‐group co‐operation, giving societies an advantage in competition with other groups—conflict with the historical and ethnographic evidence. We show that, within the framework of inclusive fitness theory, monogamous marriage can be viewed as the outcome of the strategic behaviour of males and females in the allocation of resources to the next generation. Where resources are transferred across generations, social monogamy can be advantageous if partitioning of resources among the offspring of multiple wives causes a depletion of their fitness value, and/or if females grant husbands higher fidelity in exchange for exclusive investment of resources in their offspring. This may explain why monogamous marriage prevailed among the historical societies of Eurasia: here, intensive agriculture led to scarcity of land, with depletion in the value of estates through partitioning among multiple heirs. Norms promoting high paternity were common among ancient societies in the region, and may have further facilitated the establishment of social monogamy. In line with the historical and ethnographic evidence, this suggests that monogamous marriage emerged in Eurasia following the adoption of intensive agriculture, as ownership of land became critical to productive and reproductive success.

Matrilineal kinship organization is a human social system that emphasizes interactions between matrilineal kin, i.e. individuals related only through females. The 'matrilineal puzzle' refers to the potential for tension characteristic of this social system, owing to the conflict between the interests and responsibilities of men in their roles as brother/uncle versus husband/father. From an evolutionary perspective, matrilineal kinship organization is puzzling when it diverts investment of resources from the individuals who provide the potentially highest reproductive returns. I use a game-theoretic framework to investigate a particular form of matrilineal inheritance-the transfer of property from a maternal uncle to a sororal nephew. The analysis reveals two mechanisms that may make this strategy a stable evolutionary outcome. First, a polygynous male has multiple wives, and hence multiple brothers-in-law; with matrilineal inheritance, each additional brother-in-law may transfer resources to the male's wife's offspring, thus potentially contributing to the male's inclusive fitness. Second, the husband of a polyandrous female is effectively 'sharing' paternity with other men; depending on the number of husbands, he may be better off investing in his sister's offspring. I conclude by discussing how these results address the challenges posed by the occurrence of matrilineal kinship organization.