In this paper, a novel adaptive robust control (ARC) scheme is proposed for electro-hydraulic servo systems (EHSSs) with uncertainties and disturbances. All dynamic functions in system dynamics are effectively approximated by multi-layer radial basis function neural network (RBF NN)-based approximators with online adaptive mechanisms. Moreover, neural network-based disturbance observers (NN-DOBs) are established to actively estimate and efficiently compensate for the effects of not only the matched/mismatched but also the imperfections of RBF NN-based approximators on the control system. Based on that, the nonlinear robust control law which integrates RBF NNs and NN-DOBs is synthesized via the sliding mode control (SMC) approach to guarantee the high-accuracy position tracking performance of the overall control system. Furthermore, the problem of the combination between DOBs and RBF NNs is first introduced in this paper to treat both disturbances and uncertainties in the EHSS. The stability of the recommended control mechanism is proven by using Lyapunov theory. Finally, numerical simulations with several distinct frequency levels of reference trajectory are conducted to convincingly demonstrate the effectiveness of the proposed approach.