In hydrogen recirculation side channel pumps, the motor rotor is exposed to a high-pressure mixture of steam and hydrogen, which makes hydrogen embrittlement occur in permanent magnets (PMs). A protective coating is necessary for the PMs in high-pressure hydrogen. However, in the process of sleeve interference installation, the protective coating of the PMs is easily damaged. This paper proposes two surface-mounted insert permanent magnet (SIPM) synchronous motor topologies, SIPM1 and SIPM2, in which the retaining sleeves can be eliminated and the PM protective coating is safe in the assembling process. A dovetail PM and rotor core structure is used to protect the PM with higher rotor strength without retaining the sleeve. The electromagnetic performance of the motors with different rotors, including airgap flux density, output torque, torque ripple, and energy efficiency is compared and optimized. It is concluded that the output torque of the SIPM motor can be promoted by 22.4% and torque ripple can be reduced by 2.9%, while the PM volume remains the same as that of the conventional SPM motor. At the same time, the SIPM motor can have lower harmonic contents of back electromotive force (EMF) and rotor loss compared to the SPM motor with a retaining sleeve. Furthermore, the stress of the PM is analyzed under conditions of PM glue action and failure. The proposed SIPM2 has the ability to operate safely at high-speed and high-temperature operating conditions when the PM glue fails.