We study the charge and spin dependent scattering in a set of CoFeB thin films whose crystalline order is systematically enhanced and controlled by annealing at increasingly higher temperatures. Terahertz conductivity measurements reveal that charge transport closely follows the development of the crystalline phase, with increasing structural order leading to higher conductivity. The terahertz-induced ultrafast demagnetization, driven by spin-flip scattering mediated by the spin-orbit interaction, is measurable in the pristine amorphous sample and much reduced in the sample with highest crystalline order. Surprisingly, the largest demagnetization is observed at intermediate annealing temperatures, where the enhancement in spin-flip probability is not associated with an increased charge scattering. We are able to correlate the demagnetization amplitude with the magnitude of the in-plane magnetic anisotropy, which we characterize independently, suggesting a magnetoresistance-like description of the phenomenon.