Herringbone gear transmission system is widely used in the fields of ships and aviation,etc. Loaded deformation of teeth, manufacturing and installation errors will cause GS vibration, while modification technology can reduce vibration and noise very effectively.Backlash and bearing clearance have a very important effect on GS nonlinearity. So this paper proposes a tooth surface 3d modification method based on form grinding, deduces terminal section profile equation of tooth profile modification and designs the profile of grinding wheel, finally determines its radial motion law along workpiece on axial modification,which are combined to complete 3d modification equation; Based on TCA and LTCA, obtaining LTE of herringbone gear pair and taking its minimum amplitude as optimization objective to establish 3d modification optimization model, which to obtain the optimal modification parameters by using ALO. On this basis, meshing stiffness of the optimum modified herringbone gear pair is calculated, the bending-torsion-axis coupling multi-clearance nonlinear dynamic model is established, and the nonlinear response under three internal excitations is solved by numerical method. The global vibration characteristics in the parameter domain of gear pair are studied based on bifurcation diagram and MLE, and the local vibration characteristics under specific parameters of gear pair are studied based on diagrams of time domain, phase, frequency spectrum and Poincare section. The results show:Optimized 3d modification can eliminate tooth end and tooth top edge contact and improve tooth contact performance;With the increase of input power, input speed, backlash and STE, the system has the jump, while with the increase of bearing clearance and damping ratio, there is no jump, and system vibration amplitude is significantly lower than the non-modification after being optimized modification; Compared with other parameters, the variation of input speed and STE makes system have complex and changeable bifurcation characteristics; Modification can eliminate jump, but increase chaotic interval; Modification can increase periodic window of system, make system in the alternating change of multiple periodic motion and chaotic motion, which makes system motion more regular. The study provides a basis for parameter selection and optimization of gear system with complex excitations(stiffness,impact and error).