In order to function as soft actuators, depending on their field of use, magnetorheological elastomers (MREs) must fulfill certain criteria. To name just a few, these can include rapid response to external magnetic fields, mechanical durability, mechanical strength, and/or large deformation. Of particular interest are MREs which produce macroscopic deformation for small external magnetic field variations. This work demonstrates how this can be achieved by just a small change in magnetic field orientation. To achieve this, (super)paramagnetic nickel particles of size ≈ 160 μm were embedded in a non-magnetic polydimethylsiloxan (PDMS) (661–1301 Pa) and their displacement in a stepwise rotated magnetic field (170 mT) recorded using a video microscope. Changes in particle aggregation resulting from very small variations in magnetic field orientation led to the observation of a new strongly magneto-active effect. This configuration is characterized by an interparticle distance in relation to the angle difference between magnetic field and particle axis. This causes a strong matrix deformation which in turn demonstrates hysteresis on relaxation. It is shown that the occurrence strongly depends on the particle size, particle distance, and stiffness of the matrix. Choosing the correct parameter combination, the state can be suppressed and the particle-matrix system demonstrates no displacement or hysteresis. In addition, evidences of non-negligible higher order magnetization effects are experimentally ascertained which is qualitatively in agreement with similar, already theoretically described, particle systems. Even at larger particle geometries, the new strongly magneto-active configuration is preserved and could create macroscopic deformation changes.