Aiming at the problem of stick-slip vibration caused by sudden drilling resistance torque during drilling in coal seam with gangue, the drilling tool dynamic model with two degree of freedom was established based on the interaction between the bit cutting teeth and the coal seam with gangue. So, the motion differential equation of torsional vibration of drilling tool was derived, and the torsional vibration response of drilling tool was analyzed. Taking the drilling tool with drilling depth of 300 m as an example, the response laws of angular displacement, angular velocity, resistance torque, driving torque, relative motion phase trajectory and torsional vibration of the drill bit were discussed. The results show that the drilling tool has obvious stick-slip vibration under the action of sudden drilling resistance in the process of drilling in coal seam with gangue. The angular velocity of the drill bit moves alternately between the viscous stage and the sliding stage. A stable limit cycle will appear in the phase trajectory curve of the drill bit.