To optimize the rigidity and dynamic mechanical properties of a sawing machine and improve its processing quality and stability, a design method for the sawing machine’s gearbox was proposed. First, a lightweight design of the gearbox was realized by topology optimization. Second, the sensitivity of different design variables of the new gearbox was determined via sensitivity analysis of the objective function. Finally, multi-objective optimization was used to obtain the optimal solution for the gearbox. Considering the complexity of the internal structure of the gearbox assembly and the accuracy of the numerical calculation process, a modeling method with mass points was proposed. A comparison between the numerical calculation results and the operation mode analysis revealed that the former was accurate and can be applied to the verification of the optimized gearbox. By optimizing the vibration signals before and after, and the analysis of the end face quality of the workpiece, the results revealed that the optimized gearbox has a significantly reduced amplitude under various operating conditions. In addition, the vibration stability was improved, and the end face quality of the workpiece was significantly enhanced compared to that before optimization. This study serves as a theoretical reference for multi-body dynamics modeling and optimization of machine tools, and also outlines technical solutions for high-speed stable cutting with sawing machines.