To improve the control accuracy and stability of the harmonic drive system under the influence of nonlinear friction and external disturbances, we have developed an adaptive neural backstepping control approach with friction compensation. During the design process, we employ a nonlinear observer based on a novel modified LuGre model with friction compensation. This observer effectively reduces the influence of nonlinear friction on the harmonic drive system, even under dynamic changes in the environment. Additionally, we utilize a Chebyshev neural network to approximate unknown disturbances applied to the harmonic drive system. To prevent violations of output constraints, we introduce a tangent barrier Lyapunov function. Furthermore, to address the challenges of "explosion of complexity" and poor precision associated with first-order filters in backstepping, we integrate the inverse hyperbolic sine function tracking differentiator into this controller. Finally, we employ the Lyapunov criterion to prove that all errors in the closed-loop system are uniformly bounded. Simulation results confirm the feasibility of the proposed control scheme and demonstrate better closed-loop behavior compared to that obtained using a radial basis function dynamic surface controller.INDEX TERMS Adaptive neural backstepping control, Chebyshev neural network, harmonic drive system, modified LuGre model, tangent barrier Lyapunov function.