Balance shafts are often used to improve the engine vibration characteristics of three-cylinder engines. The balance damping gear with a damping ring is an important part connecting the crankshaft and the balance shaft transmission. The stiffness characteristics of the damping ring and the unbalance of the gear have an important influence on its vibration suppression performance, but the coupled influence of the stiffness characteristics of the damping ring and the unbalanced characteristics of the vibration damping gear is unknown. In this paper, a multi-body dynamic bending–torsional coupling model of the transmission system of a three-cylinder engine with a balance damping gear is constructed considering the equivalent stiffness of the balance shaft support. Based on the fourth-order Runge–Kutta method, the influence laws of different rotational speeds, load torques, gear unbalance, radial stiffness and torsional stiffness of the damping ring on the vibration characteristics of the transmission system are obtained. The results show that the vibration amplitude increases linearly with the increase in the rotational speed and the amount of unbalance. As the load torque increases, the noise radiation of the system increases. The change in the equivalent torsional stiffness of the damping ring has little effect on the radial vibration suppression effect of the gear. As the equivalent radial stiffness of the damping ring increases, the vibration suppression rate decreases linearly. Combined with the calculation formula of damping ring stiffness, when the inner and outer diameters of the damping ring are relatively large, the vibration suppression performance decreases sharply with the increase in the thickness of the damping ring. Therefore, in order to achieve a better vibration attenuation effect, the inner to outer diameter ratio of the damping ring should be given priority in the design of the damping gear. Thus, the thickness of the design can meet the requirements of the vibration attenuation performance and a vibration attenuation of more than 90% of the radial vibration can be achieved. The model of the damping ring size and the vibration suppression effect established based on the method presented in this paper can be used to guide the design of balance damping gears.