Magnetic coil design study is carried out, for the purpose of mitigating or suppressing the edge localized modes (ELMs) in a EU DEMO reference scenario. The coil design, including both the coil geometry and the coil current requirement, is based on criteria derived from the linear, full toroidal plasma response computed by the MARS-F code (Liu et al 2000 Phys. Plasma 7 3681). With a single midplane row of coils, a coil size covering about 30°–50° poloidal angle of the torus is found to be optimal for ELM control using the n > 2 resonant magnetic perturbation (RMP) field (n is the toroidal mode number). For off-midplane coils, the coils’ poloidal location, as well as the relative toroidal phase (coil phasing) between the upper and lower rows of coils, also sensitively affects the ELM control according to the specified criteria. Assuming that the optimal coil phasing can always be straightforwardly implemented, following a simple analytic model derived from toroidal computations, it is better to place the two off-midplane rows of coils near the midplane, in order to maximize the resonant field amplitude and to have larger effects on ELMs. With the same coil current, the ex-vessel coils can be made as effective as the in-vessel coils, at the expense of increasing the ex-vessel coils’ size. This is however possible only for low-n (n = 1–3) RMP fields. With these low-n fields, and assuming 300 kAt maximal coil current, the computed plasma displacement near the X-point can meet the 10 mm level, which we use as the conservative indicator for achieving ELM mitigation in EU DEMO. The risk of partial control coil failure in EU DEMO is also assessed based on toroidal modeling, indicating that the large n = 1 sideband due to coil failure may need to be corrected, if the nominal n > 1 coil configurations are used for ELM control in EU DEMO.