The development of the machining technology is restricted by the problems of serious vibration and low system stability. The variable-pitch mill has certain vibration reduction property due to the unequal pitch angles. In this work, firstly, according to the features of the structure of the variable-pitch end mill, the cutting mechanism is analyzed, and a theoretical model of the dynamic cutting force is developed. Secondly, the cutting force coefficients of the model are determined based on the test of hardened steel for verifying the significance of the theoretical model. Then, the time-frequency characteristics and vibration reduction mechanism of variable-pitch mills are analyzed. The frequency characteristics of different types of pitch angle mills are also explored by the spectrum analysis. Finally, based on the energy and variance methods of the amplitude, the multi-objective optimization of the pitch angle is carried out, where the spectrum lines are found evenly distributed, and the optimal result is 80°-97°-100°-83°. For satisfying the machining requirements, the pitch angle of the variable-pitch end mill is reasonably selected to reduce the amplitude of the forced vibration, which can play an important role in reducing the vibration of the tool and improving its cutting stability.