The spectrum of the cavity-QED microlaser/micromaser is expected to show distinctive features of the coherent light-matter interaction, which are obscured in the conventional Schawlow-Townes linewidth theory. However, the spectral studies has been limited to resonant atom-cavity interaction so far. Here we consider the dispersive interaction in the off-resonance case, from which we uncover a quantum frequency pulling effect in the microlaser/micromaser spectrum. We present a quantum theory of the spectrum which introduces the notion of a frequency-pulling distribution associated with the photon number. In contrast to the conventional laser, periodic variation of the mean frequency pulling is observed with increasing pump parameter and it is attributed to the strong atom-cavity coupling. The pulling distribution gives rise to a spectral broadening, which can be dominant over the non-dispersive broadening addressed in the previous works. We also developed a corresponding semiclassical theory and discuss how the introduction of the frequency shift fits in with the extended quantum theory.