Let G be a locally compact group and µ a probability measure on G, which is not assumed to be absolutely continuous with respect to Haar measure. Given a unitary representation (π, H) of G, we study spectral properties of the operator π(µ) acting on H. Assume that µ is adapted and that the trivial representation 1 G is not weakly contained in the tensor product π⊗π. We show that π(µ) has a spectral gap, that is, for the spectral radius r spec (π(µ)) of π(µ), we have r spec (π(µ)) < 1. This provides a common generalization of several previously known results. Another consequence is that, if G has Kazhdan's Property (T), then r spec (π(µ)) < 1 for every unitary representation π of G without finite dimensional subrepresentations. Moreover, we give new examples of so-called identity excluding groups.