Classical Pauli particle (CPP) serves as a slow manifold, substituting the conventional guiding center dynamics. Based on the CPP, we utilize averaged vector field (AVF) method in the computations of drift orbits. Demonstrating significantly higher efficiency, this advanced method is capable of accomplishing the simulation in less than one-third time of directly computing the guiding center motion. In contrast to the CPP-based Boris algorithm, this approach inherits the advantages of the AVF method, yielding stable trajectories even achieved with a tenfold time step and reducing energy error with two orders of magnitude. By comparing these two CPP algorithms with the traditional RK4 method, numerical results indicate the remarkable performance in terms of both computational efficiency and error elimination. Moreover, we verify the properties of slow manifold integrators and successfully observe the bounce on both sides of the limiting slow manifold with initial conditions chosen off. To evaluate the practical value of the methods, we conduct simulations in non-axisymmetric perturbation magnetic fields as part of the experiments, demonstrating our CPP-based AVF method can handle simulations under complex magnetic field configurations with high accuracy, which the CPP-based Boris algorithm lacks. Through numerical experiments, we demonstrate CPP can replace guiding center dynamics in using energy-preserving algorithms for computations, providing a new, efficient, as well as stable approach for applying structure-preserving algorithms in plasma simulations.