An all-optical method of ultrafast spin rotation is put forward to precisely manipulate the polarization of relativistic charged particle beams of leptons or ions. In particular, laser-driven dense ultrashort beams are manipulated via single-shot interaction with a co-propagating moderate temporally asymmetric (frequency-chirped or subcycle THz) laser pulse. Using semi-classical numerical simulations, we find that in a temporally asymmetrical laser field, the spin rotation of a particle can be determined from the flexibly controllable phase retardation between its spin precession and momentum oscillation. An initial polarization of a proton beam can be rotated to any desired orientation (e.g., from the common transverse to the more useful longitudinal polarization) with extraordinary precision (better than 1%) in tens of femtoseconds using a feasible frequencychirped laser pulse. Moreover, the beam qualities, in terms of energy and angular divergence, can be significantly improved in the rotation process. This method has potential applications in various areas involving ultrafast spin manipulation, like laser-plasma, laser-nuclear and high-energy particle physics.