At temperatures well below the Fermi temperature $T_F$, the coupling of magnetic fluctuations to particle-hole excitations in a two-component Fermi gas leads to non-analytic corrections in the renormalized free energy and makes the transition to itinerant ferromagnetism first order. On the other hand, despite that larger symmetry often introduces larger degeneracies in the low-lying states, here we show that for a Fermi gas with SU($N > 2$)-symmetry in three space dimensions the ferromagnetic phase transition is first order in agreement with Landau's mean-field theory in its minimal formulation, which contains a cubic term in the free-energy [M. A. Cazalilla \emph{et al}. New J. of Phys. {\bf 11} 103033 (2009)]. By performing unrestricted Hartree-Fock calculations for an SU($N > 2$)-symmetric Fermi gas with short range interactions, we find the order parameter undergoes a finite jump across the transition. In addition, for SU($N > 2$) we do not observe any tri-critical point up to temperatures $T \simeq 0.5\: T_F$, for which the thermal smearing of the Fermi surface and thermal fluctuations are substantial. Beyond mean-field theory, we find that the coupling of magnetic fluctuations to particle-hole excitations makes the transition more abrupt and enhances the tendency of the gas to become fully polarized for smaller $N$ and the gas parameter $k_F a_s$. In our study, we also clarify the role of time reversal symmetry in the microscopic Hamiltonian and obtain the temperature dependence of Tan's contact. For the latter, the presence of the tri-critical point for $N = 2$ leads to a more pronounced temperature dependence around the transition than for SU($N > 2$)-symmetric gases. Although the results are obtained for a microscopic model relevant to atomic gases, they may also apply to models of solid state systems with SU($N > 2$) symmetry provided Coulomb interactions are sufficiently well screened.