This article sets up a three-dimensional model of the milling tool system, carries out modal analysis of the milling cutter with different overhangs of the tool system, and investigates the vibration patterns under different modes; conducts milling experiments on Cr12MoV mold steel, uses acceleration sensor equipment to measure the tool amplitude in the X, Y and Z directions during milling machining, and simulates and analyzes the amplitude in the three directions by response surface method, and establishes a regression model respectively, and uses ANOVA to verify the significance of the model, and finally obtains the optimal cutting parameters. A regression model was developed to verify the significance of the model using analysis of variance, and finally, the optimal cutting parameters were obtained. This study helps to further understand the interrelationship of various factors in the cutting process, provides positive theoretical analysis and practical guidance for metal cutting machining, provides strong support for improving machining efficiency and ensuring the quality of workpieces, and provides a useful reference for engineering practice in the field of precision machining.