Coatings can significantly improve the load-carrying performance of a gear surface, but how they affect the vibration characteristic of the system is an urgent issue to be solved. Taking into account the nonlinear factors like the variable mesh stiffness, friction, backlash, and transmission error, a six-degree-of-freedom spur gear transmission system with coatings is presented. Meanwhile, the finite element method is applied to acquire the time-varying mesh stiffness of the coated gear pair in the engagement process. With the support of the time-history curve, phase curve, Poincare map, and fast Fourier transform spectrum, the dynamic characteristics and the effects of the coating elastic modulus on vibration behaviors of a gear transmission system are minutely dissected by using a numerical integration approach. Numerical cases illustrate that the dynamic characteristic of a gear transmission system tends toward a one-period state under the given operating condition. They also indicate that, compared with softer coatings, stiffer ones can properly enhance the transmission performance of the coated gear pair. Numerical results are also compared with previous studies, and can establish a theoretical basis for dynamic design and vibration control of the coated gear transmission system.