Symmetries play a crucial role in understanding phases of matter and the transitions between them. Theoretical investigations of quantum models with SU(N) symmetry have provided important insights into many-body phenomena. However, these models have generally remained a theoretical idealization, since it is very difficult to exactly realize the SU(N) symmetry in conventional quantum materials for large N. Intriguingly however, in recent years, ultracold alkaline-earth-atom (AEA) quantum simulators have paved the path to realize SU(N)-symmetric many-body models, where N is tunable and can be as large as 10. This symmetry emerges due to the closed shell structure of AEAs, thereby leading to a perfect decoupling of the electronic degrees of freedom from the nuclear spin. In this work, we provide a systematic review of recent theoretical and experimental work on the many-body physics of these systems. We first discuss the thermodynamic properties and collective modes of trapped Fermi gases, highlighting the enhanced interaction effects that appear as N increases. We then discuss the properties of the SU(N) Fermi-Hubbard model, focusing on some of the major experimental achievements in this area. We conclude with a compendium highlighting some of the significant theoretical progress on SU(N) lattice models and a discussion of some exciting directions for future research.