The mammalian cerebral neocortex has a unique structure, composed of layers of different neuron types, interconnected in a stereotyped fashion. While the overall developmental program seems to be conserved, there are divergent developmental factors generating cortical diversity amongst species. In terms of cortical neuronal numbers, some of the determining factors are the size of the founder population, the duration of cortical neurogenesis, the proportion of different progenitor types, and the fine-tuned balance between self-renewing and differentiative divisions. We develop a mathematical model of neurogenesis that, accounting for these factors, aims at explaining the high diversity in neuronal numbers found across species. By framing our hypotheses in rigorous mathematical terms, we are able to identify paths of neurogenesis that match experimentally observed patterns in mouse, macaque and human. Additionally, we use our model to identify key parameters that would particularly benefit from accurate experimental investigation. We find that the timing of a switch in favor of symmetric neurogenic divisions produces the highest variation in cortical neuronal numbers. Surprisingly, assuming similar cell cycle lengths in primate progenitors, the increase in cortical neuronal numbers does not reflect a larger size of founder population, a prediction that has identified a specific need for experimental quantifications.