Diversifying agriculture by rotating a greater number of crop species in sequence is a promising practice to reduce negative impacts of crop production on the environment and maintain yields. However, it is unclear to what extent cereal yields change with crop rotation diversity and external nitrogen fertilization level over time, and which functional groups of crops provide the most yield benefit. Here, using grain yield data of small grain cereals and maize from 32 long-term (10–63 years) experiments across Europe and North America, we show that crop rotational diversity, measured as crop species diversity and functional richness, enhanced grain yields. This yield benefit increased over time. Only the yields of winter-sown small grain cereals showed a decline at the highest level of species diversity. Diversification was beneficial to all cereals with a low external nitrogen input, particularly maize, enabling a lower dependence on nitrogen fertilisers and ultimately reducing greenhouse gas emissions and nitrogen pollution. The results suggest that increasing crop functional richness rather than species diversity can be a strategy for supporting grain yields across many environments.