The semidirect drive cutting transmission system of coal cutters is prone to unstable torsional vibration when the resistance values of its driving permanent magnet synchronous motor (PMSM) are affected by changes in temperatures and tough conditions. Besides, the system has the properties of complex electromechanically coupling such as the coupling between electrical parameters and mechanical parameters. Therefore, in this study, the nonlinear torsional vibration equation was established on the basis of the Lagrange-Maxwell theory. Moreover, in light of the nonlinear dynamic bifurcation theory, the system stability was analyzed by taking the resistance value of power motor as the bifurcation parameter. In addition, the influence of subcritical bifurcation on the torsional vibration was studied by investigating the necessary and sufficient conditions for dynamic Hopf bifurcation and classifying the bifurcation types. At last, in order to suppress destabilizing oscillation induced by Hopf bifurcation, the nonlinear feedback controller was constructed, with the introduction of feedback from the motor velocity as well as the selection of voltage value on the q shaft as the controlled variable. Meanwhile, the three-order normal form and controlling parameters of the system were obtained with the aid of the multiple scales method and the harmonic balance method. In this way, the Hopf bifurcation point was transferred to control the stability of Hopf bifurcation and the amplitude of limit cycle, thus guaranteeing reliable and safe operation of the system. The numerical simulation results indicate that the designed controller boosts an ideal controlling effect.