For high-capacity flywheel energy storage system (FESS) applied in the field of wind power frequency regulation, high-power, well-performance machine and magnetic bearings are developed. However, due to the existence of axial magnetic force in this machine structure along with the uncontrollability of the magnetic bearing, the axial stability of the flywheel needs to be focused on. Firstly, a FESS with an axial flux permanent magnet synchronous machine (AFPMSM) based on soft magnetic composite (SMC) material and HALBACH axial passive magnetic bearing (PMB) structure is proposed, and its principle and structural superiority are introduced. Secondly, a three-dimensional (3D) finite element method (FEM) simulation model of the machine and bearing is established. The effects of current, air gap and bearing parameters on the rotor axial force are investigated using the 3D FEM. In addition, the relationship between current and displacement on axial force is fitted by the response surface method (RSM). The startup process and the effect of current change on displacement of the flywheel under different operating conditions are investigated by the Runge-kutta (RK) method. After that, the rotor displacement under various air gaps and bearing forces is studied to ensure that the rotor displacement is smaller than the air gap, thereby ensuring the flywheel rotor stays within a controllable range. Finally, the FESS prototype is manufactured and tested, which finally enables it to operate safely and stably.INDEX TERMS Flywheel energy storage system(FESS), axial-flux permanent magnet synchronous machine(AFPMSM), axial magnetic bearing(AMB), Runge-kutta (RK) method, Response surface modeling (RSM),axial stability.