Reinforcement learning as an effective strategy is widely utilized in optimal control. However, when updating critic–actor weight vectors based on the square of Bellman residual, it often leads to substantial computational complexity. This paper formulates a compound learning optimal backstepping control programme that can efficaciously reduce the computational burden for fractional-order predator–prey systems (FOPPS) with uncertainties. To economize resource, a reinforcement learning technology is adopted to realize the optimal control in view of neural networks under identifier–critic–actor structure. To address the computational complexity issue raised above, a simple positive definite function is proposed to update critic–actor weight vectors. Fractional-order filters are utilized to estimate virtual signals and their fractional-order derivatives for tackling the “explosion of complexity” problem existing in the conventional backstepping technology. Simultaneously, to enhance the approximation accuracy of uncertainties in FOPPS, a compound learning updating law is built by using tracking error and prediction error. In accordance with the stability analysis, the formulated scheme ensures that the output of FOPPS can track the reference signal with the expected accuracy and all signals are bounded. Eventually, a numerical simulation is presented to validate the effectiveness of the proposed control strategy.