During ultrafast demagnetization of a magnetically ordered solid, angular momentum has to be transferred between the spins, electrons, and phonons in the system on femto- and picosecond timescales. Although the intrinsic spin-transfer mechanisms are intensely debated, additional extrinsic mechanisms arising due to nanoscale heterogeneity have only recently entered the discussion. Here we use femtosecond X-ray pulses from a free-electron laser to study thin film samples with magnetic domain patterns. We observe an infrared-pump-induced change of the spin structure within the domain walls on the sub-picosecond timescale. This domain-topography-dependent contribution connects the intrinsic demagnetization process in each domain with spin-transport processes across the domain walls, demonstrating the importance of spin-dependent electron transport between differently magnetized regions as an ultrafast demagnetization channel. This pathway exists independent from structural inhomogeneities such as chemical interfaces, and gives rise to an ultrafast spatially varying response to optical pump pulses.