An automatic tensioner that consists of a torsional spring and friction damping elements is widely used in belt drive system. The relation of the applied torque versus the imposed angle of tensioner during loading and unloading processes is described as a hysteretic loop. An analytical model is established for estimating the hysteretic behavior of a tensioner, and measurements for hysteretic loop in quasi-static and dynamic excitation are carried out to validate the analytical model. Taking one engine timing belt drive system as a studying example, the method and the procedure for estimating vibration responses of the nonlinear tensioner are given. An iterative algorithm for predicting the accurate equivalent viscous damping of tensioner is carried out in analyzing a belt drive system. The vibration responses of tensioner are validated by the measurement of timing belt drive system. The developed method presented in this article is useful for predicting the hysteretic behavior of tensioner, vibration responses, and the parameters optimization of a belt drive system.