The present work investigates the non-smooth vibration characteristic and time delay feedback control of a gear pair system involving backlash and time-varying mesh stiffness. Firstly, a gear pair model with backlash non-smooth characteristic is established. Then in combination with the discontinuity mapping method, Floquet theory is presented to determine the stability and bifurcation of periodic response, and the period doubling bifurcation has been accurately predicted. Moreover, the maximal Lyapunov exponent is obtained to determine the chaos state in gear pair system which is conform to the bifurcation diagram and Poincare section. Finally, a time delay feedback is introduced to control the dynamic behaviors of the system, and numerical simulation results show that the system can be effectively controlled from chaotic motion into stable periodic motion by increasing the delay feedback gain or delay time.