The full retarded electromagnetic force experienced by swift electrons moving parallel to planar boundaries is revisited, for both metallic and dielectric targets, with special emphasis on the consequences in electron microscopy experiments. The focus is placed on the sign of the transverse force experienced by the electron beam as a function of the impact parameter. For point probes, the force is found to be always attractive. The contribution of the induced magnetic field and the causality requirements of the target dielectric response, given by the Kramers-Kronig (K-K) relations, prove to be crucial issues at small impact parameters. For spatially extended probes, repulsive forces are predicted for close trajectories, in agreement with previous works. The force experienced by the target is also explored, with the finding that in insulators, the momentum associated to Cherenkov radiation (CR) is relevant at large impact parameters.