The vibration of the feed system remains a long-standing challenge in a machine. Toward this objective, the effects of bearing stiffness, nut rigidity, coupling stiffness, screw length, screw speed and radial stiffness of worktable on the torsional vibration and axial vibration of the table are explored. In this paper, a mathematical model of multi-degree-of-freedom coupling torsional and axial vibration considering various factors is established by the second Lagrange equation, and simulation analysis and experimental verification are carried out. The effects of the variable lead and variable torsional stiffness of the nut on the torsional vibration of the workbench are analyzed. Results show that nut stiffness is the main factor affecting the vibration of the workbench. As the workbench mass, motor speed, screw length and lead increase, the axial vibration of the workbench increases. In the case of lead change, as the axial stiffness of the nut increases, the effect on the torsional vibration of the workbench is reduced. Under the variable torsional stiffness of the nut, with the increase of the lead, the influence of the lead on the torsional vibration of the workbench will increase. The experimental results provide a reference for further research and provide a theoretical basis for vibration reduction and structural optimization.