The transmission system of the cutting unit of shearer is divided into three basic components: planetary reduction form, one gear on one shaft form and a double gears on one shaft. The dynamic differential equations of three basic components are established respectively, and the volume functions of each structure are obtained. The characteristics of the internal excitation of the transmission system are analyzed, and the solution methods of the motion parameters of each component are obtained based on the harmonic balance method. Taking the parameters such as tooth number, modulus and tooth width as optimized variables, and a robust optimization method with the minimum value of multi-parameter evaluation function weighted linearly by dimensionless volume and vibration for the transmission system of the cutting unit of shearer is presented. Taking a certain type of shearer as an example, the transmission system of the cutting unit is optimized by using the presented method. After the design, the size is reduced by 5.4%, the maximum torsional acceleration of the drum is reduced by 17.8%, and the maximum torsional acceleration of the first gear is reduced by 9.6%. The results show that the design method can reduce the manufacturing cost of shearer and reduce the failure rate of the cutting unit.