Linguistic diversity is a key aspect of human population diversity and shapes much of our social and cognitive lives. To a considerable extent, the distribution of this diversity is driven by environmental factors such as climate or coast access. An unresolved question is whether the relevant factors have remained constant over time. Here, we address this question at a global scale. We approximate the difference between pre- versus post-Neolithic populations by the difference between modern hunter-gatherer versus food-producing populations. Using a novel geostatistical approach of estimating language and language family densities, we show that environmental-chiefly climate factors-have driven the language density of food-producing populations considerably more strongly than the language density of hunter-gatherer populations. Current evidence suggests that the population dynamics of modern hunter-gatherers is very similar to that of what can be reconstructed from the Palaeolithic record. Based on this, we cautiously infer that the impact of environmental factors on language densities underwent a substantial change with the transition to agriculture. After this transition, the environmental impact on language diversity in food-producing populations has remained relatively stable since it can also be detected-albeit in slightly weaker form-in models that capture the reduced linguistic diversity during large-scale language spreads in the Mid-Holocene.