Gears serve as vital elements in high-efficiency mechanical transmissions, playing a pivotal role in vehicles. Employing gear modification has the capacity to significantly enhance the dynamic properties of the Planetary Gear Transmission (PGT) systems. In this study, the investigation has been directed toward a three-stage PGT model system with a comprehensive consideration of various vibration-related factors that affect the dynamic behavior of the system. These factors are encompassed by Time Varying Mesh Stiffness (TVMS), torsion and bending forces experienced by connecting shafts, dynamic Transmission Error (TE), impact of gear meshing eccentric load forces, inherent flexibility in the supporting shaft structure, and influence of torque and power fluctuations originating from a multi-cylinder engine. The estimation of TVMS is carried out for both external-external and external-internal teeth meshing pairs, utilizing the Potential Energy Method (PEM). Subsequently, this study employs the lumped parameter method to establish a model with a specific objective of investigating the lateral-torsional-coupling behavior within adopted PGT model system. The lateral vibration displacement characteristics of the PGT model system is deeply analyzed, revealing the influence of speed on lateral vibration and displacement frequency domain characteristics through a 3D waterfall diagram. Furthermore, a Tooth Profile Modification (TPM) method is applied utilizing two distinct parameters namely [Formula: see text] TPM parameter and [Formula: see text] TPM parameter. The objective is to mitigate vibrations and evaluate the performance of the adopted PGT model after employing each TPM parameter through both TVMS and Root Mean Square (RMS).