We consider nonlinear dynamics of a single vortex in a superconductor in a strong rf magnetic field B0 sin ωt. Using the London theory, we calculate the dissipated power Q(B0, ω), and the transient time scales of vortex motion for the linear Bardeen-Stephen viscous drag force, which results in unphysically high vortex velocities during vortex penetration through the oscillating surface barrier. It is shown that penetration of a single vortex through the ac surface barrier always involves penetration of an antivortex and the subsequent annihilation of the vortex antivortex pairs. Using the nonlinear Larkin-Ovchinnikov (LO) viscous drag force at higher vortex velocities v(t) results in a jump-wise vortex penetration through the surface barrier and a significant increase of the dissipated power. We calculate the effect of dissipation on nonlinear vortex viscosity η(v) and the rf vortex dynamics and show that it can also result in the LO-type behavior, instabilities, and thermal localization of penetrating vortex channels. We propose a thermal feedback model of η(v), which not only results in the LO dependence of η(v) for a steady-state motion, but also takes into account retardation of temperature field around rapidly accelerating vortex, and a long-range interaction with the surface. We also address the effect of pinning on the nonlinear rf vortex dynamics and the effect of trapped magnetic flux on the surface resistance Rs calculated as a function or rf frequency and field. It is shown that trapped flux can result in a temperature-independent residual resistance Ri at low T , and a hysteretic low-field dependence of Ri(B0), which can decrease as B0 is increased, reaching a minimum at B0 much smaller than the thermodynamic critical field Bc. We propose that cycling of rf field can reduce Ri due to rf annealing of magnetic flux which is pumped out by rf field from the thin surface layer of the order of the London penetration depth.