Aiming at the influence of noise, vibration, and force on machining efficiency and machining quality in the process of milling, this paper proposed an improved particle swarm optimization algorithm. The algorithm could optimize the multi-objective and multi-parameter integration model and obtain the optimal milling parameters under the specified milling conditions, so as to reduce milling noise, vibration, and force and improve machining efficiency. First, based on the theory of all factor experiment, the milling experiment of TC4 titanium alloy was carried out, and the relevant milling noise, vibration, and force signals were obtained. Then, based on the response surface analysis method, the influence of milling parameters on noise, vibration, and force was studied. Finally, a multi-objective optimization model based on maximum metal removal rate, minimum noise, minimum vibration, and minimum force was established, and the improved particle swarm optimization algorithm was used to solve the optimal milling parameters. The result shows that when the milling speed is 24.9588 m/min, the feed speed is 9.9324 mm/min, and the milling depth is 1 mm, the optimal milling state within the constraint conditions can be achieved.