The nonlinear torsional vibration and instability oscillation caused by nonlinear damping in the shearer electromechanical coupling cutting transmission system in shearer driven by the permanent magnet synchronous motor (PMSM) is investigated in this paper. The electromechanical coupling transmission system in the shearer is equivalent to a concentrated mass model for the purpose of establishing the system dynamic model by the Lagrange–Maxwell equation. Then, the Routh–Hurwitz criterion is used to determine the torsional vibration critical point and stability region for the Hopf bifurcation for the cutting transmission system. According to the Routh–Hurwitz stability criterion, the Hopf bifurcation type and the effect of the supercritical Hopf bifurcation in the torsional vibration of the cutting transmission system are analyzed. Furthermore, based on the washout filter, the Hopf bifurcation controller is designed for suppressing the transmission system’s large vibration amplitude and unstable oscillation. In addition, the influences of the linear gain and nonlinear gain on the bifurcation point and the limit cycle amplitude are discussed. Finally, the numerical simulation results indicate the effectiveness of the designed controller. The research achievements can provide a theoretical basis for design or optimize the cutting transmission system of high-reliability shearer driven by PMSM.