Hydraulic servo actuator always suffers from various disturbance and uncertainties, which makes it difficult to design a higher performance controller. In this paper, an integral nonsingular terminal sliding mode controller based on extended state observer (ESO-INTSM) is proposed to improve the robust performance of hydraulic servo actuator. The ESO is designed to estimate not only the parametric uncertainties but also the model disturbance. Based on the observed states of ESO, the proposed controllers could enable hydraulic servo actuator to track the desired motion trajectories. The stability of the synthesized controller is proved via Lyapunov analysis, which is very important for high-accuracy tracking control of hydraulic servo actuator. Simulation and experimental results demonstrate that the proposed control strategy can effectively attenuate the adverse influence caused by the uncertainties and apparently improve the tracking accuracy.