Ferroic materials are characterized by the presence of electric/magnetic dipoles that can be irreversibly re-oriented under sufficiently strong loading. Upon poling, an initially random distribution of dipoles on the microscopic domain scale results in a non-vanishing state of remanent polarization/magnetization on the macroscopic level. The remanent switching of dipoles constitutes the "smartness" in materials and their usage both as sensors and as actuators. Once being poled, material properties are typically assumed to be constant and uni-directional in engineering problems. To extend the operational range of ferroic transducers and open perspectives for novel applications, we seek for an accurate understanding of the evolution of the polarization/magnetization, which requires both physical and geometric non-linearities to be included in the modeling. To describe irreversible changes of the remanent state, we transfer phenomenological concepts and algorithms of associative elasto-plasticity to the field of electro-magneto-mechanics. To describe the dissipative response in a thermodynamically consistent manner, the principle of maximum dissipation is adopted. As in elasto-plasticity, constitutive equations for dissipative internal forces that drive the evolution of the remanent polarization/magnetization then follow as associated flow rules. To simplify the algorithmic treatment, we introduce the notion of dissipation functions, by which the constrained optimization problem is converted into an unconstrained problem, which can be efficiently solved by standard means. The vectors of remanent polarization/magnetization enter the model as additional internal variables. The constitutive response is governed by thermodynamic potentials, i.e., the free energy and dissipation functions. We derive a (mixed) variational formulation, identify appropriate function spaces for the dependent fields involved, and introduce a finite-element discretization. Representative numerical examples demonstrate the efficacy of the framework in electro-magneto-mechanically coupled problems.