Aiming at the reliability optimization design problem of two-grade helical cylindrical gear transmission system, the UG parameterized model based on expression and ADAMS dynamic simulation model under real working conditions are integrated in ISIGHT platform, and the distribution law of peak contact force on the tooth surface of helical gear pair in the transmission system at different meshing times is obtained. Considering the uncertainty of material properties, load conditions and geometric parameters, the sample set was generated according to the orthogonal experimental design method, and the Kriging approximation model was established. On this basis, the contact strength reliability of helical gear was analyzed. The mathematical model of meshing efficiency reliability optimization of helical gear pair was established and solved by Hooke-Jeeves Direct Search method. The results show that the meshing efficiency and reliability of the optimized helical gear pair are improved, and the peak value of the contact force is obviously reduced.