The position accuracy of the third mirror's Z axis will directly affect the measurement accuracy of the optical telescope, so it is necessary to improve the position accuracy. In order to improve the control performances of the third mirror's Z-axis of a large optical telescope, this paper proposes a novel control algorithm based on hyperbolic tangent function (tanh). The proposed algorithm is used to replace the traditional integral separated proportional-integral (IS-PI) controller which is widely used in industry servo control system. Firstly, a mathematical model of the third mirror's Z-axis is established. Secondly, a controller based on tanh is proposed, and the characteristics and stability of the algorithm are analyzed. Thirdly, the parameters tuning method of the proposed algorithm is given. The simulation results show that the proposed algorithm has higher tracking accuracy than IS-PI controller. Finally, experiments are carried out on a large optical telescope's Z axis. In the experiments, IS-PI controller and tanh controller are compared, and the results show that the control performances are greatly improved.INDEX TERMS high precision position control, motor control, hyperbolic tangent function, optical telescope, Integral separation proportional integral.