By considering the lubricant in gear system, one degree-of-freedom model is set up which incorporates the pinion's speed and the drag torque as the excitation sources. By introducing a permissible error (ε), a new computational algorithm using double-changed time steps is proposed in order to reduce the ill-conditioning arising from the numerical stiffness of the gear system and validated by comparison with Runge-Kutta-Fehlberg integration scheme. Then, the influences of the lubricant on the vibration of the gear system are analyzed. The results obtained in this paper indicate that the proposed numerical algorithm not only improves the accuracy of the solution, but also accelerates the calculation speed of the whole system. And according to the collision feature, the contributions of the lubricant on the system are totally different with different pinion's speed and drag torque. Next, by introducing the proposed computational algorithm into the Floquet theory, the stability analyses of the gear system are investigated under the different excitation sources, which demonstrates that the excitation sources significantly affect the operating instability regions. In practice, particular instabilities can be minimized by the proper selection of pinion's speed and drag torque, which can be adjusted according to the working requirements in advance.