The operators of overhead traveling cranes experience discomfort as a result of the vibrations of crane structures. These vibrations are produced by defects in the rails on which the cranes move. To improve the comfort of operators, a nine-degree-of-freedom (nine-DOF) mathematical model of a “human–crane–rail” system was constructed. Based on the theoretical guidance provided in ISO 2631-1, an annoyance rate model was established, and quantization results were determined. A dynamic optimization design method for overhead traveling cranes is proposed. A particle swarm optimization (PSO) algorithm was used to optimize the crane structural design, with the structure parameters as the basic variables, the annoyance rate model as the objective function, and the acceleration amplitude and displacement amplitude of the crane as the constraint conditions. The proposed model and method were used to optimize the design of a double-girder 100 t–28.5 m casting crane, and the optimal parameters are obtained. The results show that optimization decreases the human annoyance rate from 28.3% to 9.8% and the root mean square of the weighted acceleration of human vibration from 0.59 m/s2to 0.38 m/s2. These results demonstrate the effectiveness and practical applicability of the models and method proposed in this paper.