In this study, a fuzzy wavelet neural network with the improved Levenberg–Marquardt algorithm (FWNN-LM) is proposed to conquer nonlinearity and uncertain disturbance problems in the AC servo system. First of all, use the particle swarm optimization algorithm based on Levenberg–Marquardt (LM) to optimize parameters in the FWNN controller. Second, the potentiality of fuzzy rules (PFR) method is developed to optimize the structure of the FWNN by error reduction ratio (ERR). Furthermore, stability of FWNN-LM is proved by the Lyapunov method. Finally, simulation and prototype test results show that this method can improve the accuracy and robustness of the system in presence of load disturbances and parameter perturbations.