Ferrogels, i.e., hydrogels loaded with magnetic nanoparticles, have the ability to deform in external magnetic fields. The precise shape of deformation and the alignment of the gel in the field, however, depend on the interplay of several factors. In this paper, we introduce a coarse-grained simulation model, which takes into account the configuration of magnetic particles in the gel, the sample shape, and aspects of the polymer network topology. We use this model to show that in gels with an isotropic microstructure, an external magnetic field reduces clustering, while this is not the case for uniaxial gels, in which the particle configuration is anisotropic due to the presence of a magnetic field during cross-linking. For ellipsoidal gels, we find that a uniaxial microstructure additionally can override the deformation and alignment expected as indicated the demagnetization energy. Finally, we examine to what degree gels with different network topologies maintain the particle microstructure "frozen in" during the cross-linking process.