Dynamic stabilization operation can effectively improve the mechanical performance of ballast bed by using stabilizer. To improve the stabilization effect, a rigid-flexible coupling dynamics simulation model of the stabilizer-ballast bed system was established according to the actual structure of the stabilizer, and the effect of different operation parameters of stabilizer on the stabilization operation was analyzed. The simulation results showed that: the maximum value of lateral displacement response of the sleeper decreased with the increase in clamping wheel force normal angle, excitation force frequency and downforce; and increased with the increase in excitation force amplitude. The optimized operation parameters were obtained as follows: clamping wheel force normal angle 0°, excitation force frequency 18 Hz, excitation force amplitude 180 kN, downforce 60 kN. A vibration test bench simulating the stabilization operation process was built according to the working principle of the stabilizer, and the reliability of the simulation was verified by testing.