Afro-Eurasian monkeys originated in the Miocene and are the most species-rich modern primate family. Molecular and fossil data have provided considerable insight into their evolutionary divergence, but we know considerably less about the evolutionary processes that underlie these differences. Here, we apply tests developed from quantitative genetics theory to a large (n > 3000) cranio-mandibular morphometric dataset, investigating the relative importance of adaptation (natural selection) and neutral processes (genetic drift) in shaping diversity at different taxonomic levels, an approach applied previously to monkeys of the Americas, apes, hominins, and other vertebrate taxa. Results indicate that natural selection, particularly for differences in size, plays a significant role in diversifying Afro-Eurasian monkeys as a whole. However, drift appears to better explain skull divergence within the subfamily Colobinae, and in particular the African colobine clade, likely due to habitat fragmentation. Small and declining population sizes make it likely that drift will continue in this taxon, with potentially dire implications for genetic diversity and future resilience in the face of environmental change. For the other taxa, many of whom also have decreasing populations and are threatened, understanding adaptive pressures similarly helps identify relative vulnerability and may assist with prioritising scarce conservation resources.