The discovery of over 50 planets around evolved stars and more than 35 debris discs orbiting white dwarfs highlight the increasing need to understand small body evolution around both early and asymptotic giant branch (GB) stars. Pebbles and asteroids are susceptible to strong accelerations from the intense luminosity and winds of GB stars. Here, we establish equations that can model time-varying GB stellar radiation, wind drag and mass loss. We derive the complete three-dimensional equations of motion in orbital elements due to (1) the Epstein and Stokes regimes of stellar wind drag, (2) Poynting-Robertson drag, and (3) the Yarkovsky drift with seasonal and diurnal components. We prove through averaging that the potential secular eccentricity and inclination excitation due to Yarkovsky drift can exceed that from PoyntingRobertson drag and radiation pressure by at least three orders of magnitude, possibly flinging asteroids which survive YORP spin-up into a widely dispersed cloud around the resulting white dwarf. The GB Yarkovsky effect alone may change an asteroid's orbital eccentricity by ten per cent in just one Myr. Damping perturbations from stellar wind drag can be just as extreme, but are strongly dependent on the highly uncertain local gas density and mean free path length. We conclude that GB radiative and wind effects must be considered when modelling the post-main-sequence evolution of bodies smaller than about 1000 km.