Two-level quantum systems, qubits, are not the only basis for quantum computation. Advantages exist in using qudits, d-level quantum systems, as the basic carrier of quantum information. We show that color codes, a class of topological quantum codes with remarkable transversality properties, can be generalized to the qudit paradigm. In recent developments it was found that in three spatial dimensions a qubit color code can support a transversal non-Clifford gate and that in higher spatial dimensions additional non-Clifford gates can be found, saturating Bravyi and König's bound [S. Bravyi and R. König, Phys. Rev. Lett. 111, 170502 (2013)]. Furthermore, by using gauge fixing techniques, an effective set of Clifford gates can be achieved, removing the need for state distillation. We show that the qudit color code can support the qudit analogs of these gates and also show that in higher spatial dimensions a color code can support a phase gate from higher levels of the Clifford hierarchy that can be proven to saturate Bravyi and König's bound in all but a finite number of special cases. The methodology used is a generalization of Bravyi and Haah's method of triorthogonal matrices [S. Bravyi and J. Haah, Phys. Rev. A 86, 052329 (2012)], which may be of independent interest. For completeness, we show explicitly that the qudit color codes generalize to gauge color codes and share many of the favorable properties of their qubit counterparts.