It is widely accepted that the magnetic state of a ferromagnetic material may be irreversibly altered by mechanical loading due to magnetoelastic effects. A novel standardized nondestructive testing (NDT) technique uses weak magnetic stray fields, which are assumed to arise from inhomogeneous deformation, for structural health monitoring (i.e., for detection and assessment of damage). However, the mechanical and microstructural complexity of damage has hitherto only been insufficiently considered. The aim of this study is to discuss the phenomenon of inhomogeneous “self-magnetization” of a polycrystalline ferromagnetic material under inhomogeneous deformation experimentally and with stronger material-mechanical focus. To this end, notched specimens were elastically and plastically deformed. Surface magnetic states were measured by a three-axis giant magnetoresistant (GMR) sensor and were compared with strain field (digital image correlation) and optical topography measurements. It is demonstrated that the stray fields do not solely form due to magnetoelastic effects. Instead, inhomogeneous plastic deformation causes topography, which is one of the main origins for the magnetic stray field formation. Additionally, if not considered, topography may falsify the magnetic signals due to variable lift-off values. The correlation of magnetic vector components with mechanical tensors, particularly for multiaxial stress/strain states and inhomogeneous elastic-plastic deformations remains an issue.