Magneto-active polymers (MAPs), composed of polymer matrices and magnetic filler particles, are smart materials that deform quickly in an external magnetic field. The ability to produce large deformation of MAPs makes these materials promising for actuators and sensors. Due to the viscoelasticity of the polymer matrices, MAPs usually demonstrate ratedependent dynamic properties. However, very few models of coupled magnetic field and viscoelasticity in MAPs exist in the literature, and even fewer are capable of reliable predictions. Starting from nonequilibrium thermodynamics, a field theory is developed to fully couple the finite-deformation viscoelasticity and magnetostatics of MAPs. The theory provides a guideline for experimental characterization of MAPs, and most material laws are readily applicable in this framework. A specific material model is prescribed for an idealized MAP. As demonstrations, numerical examples are implemented on the responses of the MAP in response to both uniform and nonuniform magnetic fields. In the nonviscous limit, our theory recovers a model for elastic MAPs, and is capable of capturing instability phenomena observed in the experiments.