A novel fiber grating vibration demodulation system, based on 2×2 and 3×3 couplers, is designed. Based on the phase unwrapping algorithm, the three-way asymmetrical output of the 3×3 coupler and demodulation state characteristics of the system when the vibration signal includes high harmonic waves are analyzed in simulations. The result shows that when the three-way output is asymmetrical, the maximum deviation of demodulation signal is 1.625%, and when the vibration signal includes high harmonic waves, the maximum deviation of demodulation signal is 0.9%. The corresponding experiment is conducted. The experimental result shows that the dynamic resolution of the system is 25.22Hz / n when the vibration pitch is 5.5 Hz.