Coherent microwave radar utilizes the direct relationship between the orbital wave velocity and the wave height spectrum to retrieve wave parameters. However, due to the broken wave and its evolution, the "Group-Line" is introduced in the radar-obtained wavenumber-frequency spectrum, and the dominant wave energy is reduced. Many methods have proposed to remove the "Group-Line" in the wavenumber-frequency spectrum, but these methods can only remove low-frequency energy and fail to compensate for the dominant wave energy. To solve this problem, a method for wave parameter inversion using the quasibinary variational mode decomposition (QB-VMD) is proposed. Firstly, the QB-VMD decomposes the spatial-temporal radial velocity series, and a series of modes are obtained, including the "Group-Line" and dominant wave modes. Secondly, the "Group-Line" mode is discarded, and the dominant wave mode is compensated appropriately to reconstruct the spatial-temporal radial velocity series. Finally, the wave parameters are retrieved according to the reconstructed spatial-temporal radial velocity series. The proposed method is verified by simulation. In addition, this paper uses on 8-day data set collected by the coherent Sband radar deployed at the Beishuang island for analysis. The significant wave height and mean wave period are retrieved from the data set. The wave parameters estimated by the proposed method are compared with the buoy results. The correlation coefficients are 0.97 and 0.80, and the root means square errors are 0.12 m and 0.49 s, respectively. The results show that the proposed method can invert wave parameters using the coherent S-band radar with a reasonable performance.