Electromagnetic devices (EMDs) containing permanent magnets are common devices showing low power consumption , small size, and high sensitivity. In batch production of EMDs there is a robustness problem linked to the migration of the working point of permanent magnets and to their manufacturing uncertainties. In turn, the design optimization for EMDs has attracted large attention. Robust design optimization (RDO) is one of the most effective methods to boost quality and robustness of EMDs. However, state-of-art RDO methods ignore the various uncertainties from manufacture processing, production environment, and working point migration of permanent magnets. To address those problems, this study presents an efficient and universal method based on the working point migration and various uncertainties of output characteristics for electromagnetic devices. At first, we introduce a two-dimensional factor design scheme considering the working point migration and manufacturing conditions. Then, a robustness assessment measure suitable for different optimization types is proposed. On these bases, the optimization model is established, considering the permanent working point migration for magnets, manufacturing tolerances and robustness constraints, and a particle swarm algorithm is used to obtain the solution. Finally, the effectiveness of the method is verified by a case study involving a specific type of electromagnetic device with permanent magnets.