This paper focuses on optimization of the geometrical parameters of peripheral milling tools by taking into account the dynamic effect. A substructure synthesis technique is used to calculate the frequency response function of the tool point, which is adopted to determine the stability lobe diagram. Based on the Taguchi design method, simulations are first conducted for varying combinations of tool overhang length, helix angle, and teeth number. The optimal geometrical parameters of the tool are determined through an orthogonal analysis of the maximum axial depth of cut, which is obtained from the predicted stability lobe diagram. It was found that the sequence of every factor used to determine the optimal tool geometrical parameters is the tool overhang length, teeth number, and helix angle. Finally, a series of experiments were carried out as a parameter study to determine the influence of the tool overhang length, helix angle, and teeth number on the cutting stability of a mill. The same conclusion as that obtained through the simulation was observed.