Aiming at the nonlinear low-speed-jitter (LSJ) vibration suppression for a yaw system of a megawatt wind turbine, a kinematics mechanism of the yaw system is investigated from the perspective of tribology, and a kinematics model of the yaw system based on an equilibrium position is established. On the basis of the dynamic modeling of the yaw system, a nonlinear mathematical model of the LSJ system is deduced. Based on the two lead motors’ driving of the conventional yaw motion, an innovative design with a special installation of two auxiliary motors for yaw transmission is carried out, which is integrated with a matching centralized lubrication system (CLS). Based on open-loop proportional-derivative (PD) control and the iterative learning control methods of the time-varying continuous system, the stability control and jitter amplitude suppression of the yaw system are realized by using a combined driving torque provided by the lead and auxiliary gears. From the stability and convergence of the time-domain response and the convergence of the iterative error, the effectiveness of the iterative learning control method with the PD-based regulation is verified, and its advantages for engineering applications are shown based on the algorithm solver improvement. The feasibility of the physical realization and engineering application of the control methodology is verified by using controller-hardware-in-the-loop (C-HITL) simulation technology.