We report systematic calculations of the low-temperature diffusion thermopower of Al-based dilute alloys with 3d and 4sp impurities, by solving self-consistently the linearized Boltzmann equation. The impurity scattering is described by the phase shifts obtained from self-consistent local-density-functional impurity-in-jellium calculations. Moreover, the influence of the full anisotropy of the Al Fermi surface on the scattering process is taken into account within the on-Fermi-sphere approximation. Our results explain successfully the experimentally measured variations in the thermoelectric power, except for Mn impurities. In this case, the presence of a narrow many-body resonance at the Fermi level in the localized spin-fluctuations regime seems to be responsible for the large negative value of thermopower observed.