In view of the numerical instability and low accuracy of the traditional transfer matrix method in solving the high-order critical speed of the rotor system, a new idea of incorporating the finite element method into the transfer matrix is proposed. Based on the variational principle, the transfer symplectic matrix of gyro rotors suitable for all kinds of boundary conditions and supporting conditions under the Hamilton system is derived by introducing dual variables. To verify the proposed method in rotor critical speed, a numerical analysis is adopted. The simulation experiment results show that, in the calculation of high-order critical speed, especially when exceeding the sixth critical speed, the numerical accuracy of the transfer symplectic matrix method is obviously better than that of the reference method. The relative errors between the numerical solution and the exact solution are 0.0347% and 0.2228%, respectively, at the sixth critical speed. The numerical example indicates the feasibility and superiority of the method, which provides the basis for the optimal design of the rotor system.