Magnetic elastomers (MEs) respond to an applied magnetic field through magnetomechanical coupling, where the mechanical properties of the MEs change with magnetic field strength. These phenomena have been mostly studied under homogenous magnetic fields due to the simplicity. In this work, the effects of the magnetic field gradient on the mechanical properties and the response of the MEs was examined. MEs are made by embedding carbonyl iron microparticles (CI) into a polydimethylsiloxane (PDMS) matrix, which is later rendered porous. The influence of the CI concentration was investigated by manipulating four different samples with CI/PDMS weight ratios of 0.2, 0.6, 1.0, and 1.4. An analytical method was proposed to further understand the interactions of the magnetic field gradient and the material’s response. The proposed theory was later verified with experimental results from compression tests in the presence of different magnetic fields. The proposed theoretical framework and experimental methods can be used to improve the design of MEs in the future.