The drilling mud shows strong non-Newtonian rheology due to mixing a large number of cuttings at well bottom during drilling, which has great influences on the vibration behavior of the drill string. To investigate this phenomenon, an axial-lateral-torsion coupling (ALTC) nonlinear dynamic model based on Hamilton's principle was established, taking into account the effect of drilling fluid viscous damping, Herschel-Bulkley non-Newtonian rheological (HBNR) damping, bit-rock interaction and drillstring-borehole contact. In the model, a Rayleigh damping matrix calculation method was proposed to describe the influence of drilling fluid on the damping of ALTC vibration, and the Newmark-β method was used to solved the nonlinear discrete equations of the system. The validity of the model was verified through the simulation data and the field measurement data. Using the nonlinear model, the effect of drilling fluid rheology on ALTC vibration response of rotary drill string was investigated. It was found that compared with viscous damping, HBNR damping can significantly suppress the axial, lateral and torsional vibration of drill string, maintain the stability of wellbore diameter and improve ROP. Both damping models indicate that lateral vibration signals are difficult to transmit to the wellhead, while axial vibration and torsional vibration signals can be observed near the wellhead. The results can provide a theoretically guidance for suppressing the drill string vibration and reducing the risk of premature failure of drill string.