Based on mode coupling theory and numerical simulation, a comprehensive systematical study of highfrequency characteristics in a 170GHz smooth walled coaxial gyrotron cavity operating with mode is presented. In particular, the change of eigenvalues of operating mode, the coupling coefficients between different modes, the influence of mode coupling on diffraction quality factor and resonant frequency, the influence of single-mode and multi-mode coupling on high frequency field distribution as well as the phase of operating mode are investigated. The results show that the influence of the TM-type parasitic modes is extremely small, and can be ignored, but TE-type parasitic modes have some influences on diffraction quality factor of operating mode ,that on resonant frequency of operating mode can be neglected.