We study the transient motion of a colloidal particle actively dragged by an optical trap through different viscoelastic fluids (wormlike micelles, polymer solutions, and entangled λ-phage DNA). We observe that, after sudden removal of the moving trap, the particle recoils due to the recovery of the deformed fluid microstructure. We find that the transient dynamics of the particle proceeds via a double-exponential relaxation, whose relaxation times remain independent of the initial particle velocity whereas their amplitudes strongly depend on it. While the fastest relaxation mirrors the viscous damping of the particle by the solvent, the slow relaxation results from the recovery of the strained viscoelastic matrix. We show that this transient information, which has no counterpart in Newtonian fluids, can be exploited to investigate linear and nonlinear rheological properties of the embedding fluid, thus providing a novel method to perform transient rheology at the micron-scale.