Anisotropic magnetoresistance (AMR) is a ubiquitous and versatile probe of ferro-, ferri-and antiferromagnetic order in contemporary spintronics research. Its origins are usually ascribed to spin-dependent electron scattering, whereas scattering-independent (intrinsic) contributions are neglected. Here, we measure AMR of the standard ferromagnets Co, Ni, Ni 81 Fe 19 and Ni 50 Fe 50 over the very wide frequency range from DC to 28 THz, which covers the regimes of both diffusive and ballistic transport. Analysis of the broadband response based on Boltzmann-Drude theory reveals that the AMR not only has the familiar scattering-dependent contribution, but also contains a sizeable scattering-independent component. In polycrystalline Co, this component is frequency-independent up to 28 THz and amounts to more than 2/3 of the AMR contrast of 1%, thereby making it interesting for applications in terahertz spintronics and terahertz photonics. Our results show that broadband terahertz electromagnetic pulses provide new insights into magneto-transport phenomena of magnetic materials on ultrafast time scales.