The definitions of the n th Gauss sum and the associated n th central charge are introduced for premodular categories C and n ∈ Z. We first derive an expression of the n th Gauss sum of a modular category C, for any integer n coprime to the order of the Tmatrix of C, in terms of the first Gauss sum, the global dimension, the twist and their Galois conjugates. As a consequence, we show for these n, the higher Gauss sums are d-numbers and the associated central charges are roots of unity. In particular, if C is the Drinfeld center of a spherical fusion category, then these higher central charges are 1. We obtain another expression of higher Gauss sums for de-equivariantization and local module constructions of appropriate premodular and modular categories. These expressions are then applied to prove the Witt invariance of higher central charges for pseudounitary modular categories.