The difference in fuel consumption of wheel loaders can be more than 30% according to different shoveling trajectories for shoveling operations, and the optimization of shoveling trajectories is an important way to reduce the fuel consumption of shoveling operations. The existing shoveling trajectory optimization method is mainly through theoretical calculation and simulation analysis, which cannot fully consider the high randomness and complexity of the shoveling process. It is difficult to achieve the desired optimization effect. Therefore, this paper takes the actual shoveling operation data as the basis. The factors that have a high impact on the fuel consumption of shoveling are screened out through Kernel Principal Component Analysis. Moreover, the mathematical model of fuel consumption of shoveling operation is established by Support Vector Machine and combined with the Improved Particle Swarm Optimization algorithm to optimize the shoveling trajectory. To demonstrate the generalization ability of the model, two materials, gravel, and sand, are selected. Meanwhile, the influence of different engine speeds on the shoveling operation is considered. We optimize the shoveling trajectories for three different engine speeds. The optimized trajectories are verified and compared with the sample data and manually controlled shoveling data. The results show that the optimized trajectory can reduce the fuel consumption of shoveling operation by 27.66% and 24.34% compared with the manually controlled shoveling of gravel and sand, respectively. This study provides guidance for the energy-efficient operation of wheel loaders.