As a typical electromechanically coupled system, the super-high-speed permanent magnet synchronous motor (PMSM)-driven compressor always exhibits complex dynamic behavior, affecting the comprehensive performance of the fuel cell system. Based on this, this paper takes electromagnetic and load excitations into account and establishes a mathematical model of the super-high-speed PMSMdriven compressor. Then, the corresponding simulation is carried out, revealing that according to different causes and manifestations, the system gradually exhibits amplitude instability and frequency instability. Considering the stiffness softening effect, the effect of the torsional stiffness and damping coefficient on the dynamic characteristics under different forms of instability is obtained. Using the Routh-Hurwitz criterion and Melnikov theory, a damping optimization methodology is given. The results show that under the condition of amplitude instability, damping reduction and stiffness softening lead to a greater resonant amplitude and a wider resonance region. Under the condition of frequency instability, the system becomes chaotic via periodic-doubling bifurcation with the decrease of damping, and the decrease of torsional stiffness increases the damping required to maintain the stability. INDEX TERMS super-high-speed permanent magnet synchronous motor, fuel cell system, stiffness softening, damping optimization