The problem of mode competition can present a major hurdle for gyrotron to achieve stable and efficient operation, especially when the gyrotron operates at electron cyclotron harmonics. Therefore, an effective theoretical investigation of the mode competition is an essential first step to the implementation of gyrotron. In this paper, the nonlinear dynamics of the mode competition in the terahertz gyrotron operating at high-order mode are studied using a multifrequency time-dependent model. Based on the model, the simulations of mode competition in the designed gyrotron are presented by the corresponding code. The influence of the velocity spread on the mode competition is also studied. The simulations tracking five competing modes show that stable excitation of the second harmonic oscillation can be realized with a proper choice of the operation parameters. The results show that a single-mode second harmonic radiation with a power of over 120 kW at a frequency of 0.4 THz could be achieved.