A hydraulic shaking table has obvious frictional characteristics that will result in the deterioration of test waveforms and affect test accuracy. In this study, the influence of nonlinear friction for a hydraulic shaking table test system is analyzed briefly. For dynamic friction parameters and unknown system load characteristics, a mathematical model is developed based on the LuGre friction model, and double nonlinear observers are constructed to estimate average deflection. A backstepping integral control method is proposed to compensate the nonlinear friction and estimate the unknown load disturbances. The stability of the closed-loop system is proved by the Lyapunov function. In the test, an iterative control algorithm is utilized to obtain the system inverse transfer function. The experimental results show that the proposed method can reduce the nonlinear friction of the hydraulic servo system, and improve the control accuracy of the hydraulic shaking table. In addition, high identification accuracy for system impedance is also achieved. Compared to no friction compensating method, the acceleration waveform distortion is reduced from 19 % to 5 % and the peak error is reduced from 14 % to 4.2 % by the proposed method, and the deviation of power spectral density is also reduced effectively.