The measured frequency response functions (FRFs) in the modal test are usually contaminated with noise that significantly affects the modal parameter identification. In this paper, a modal peak-based Hankel-SVD (MPHSVD) method is proposed to eliminate the noise contaminated in the measured FRFs in order to improve the accuracy of the identification of modal parameters. This method is divided into four steps. Firstly, the measured FRF signal is transferred to the impulse response function (IRF), and the Hankel-SVD method that works better in the time domain rather than in the frequency domain is further applied for the decomposition of component signals. Secondly, the iteration of the component signal accumulation is conducted to select the component signals that cover the concerned modal features, but some component signals of the residue noise may also be selected. Thirdly, another iteration considering the narrow frequency bands near the modal peak frequencies is conducted to further eliminate the residue noise and get the noise-reduced FRF signal. Finally, the modal identification method is conducted on the noise-reduced FRF to extract the modal parameters. A simulation of the FRF of a flat plate artificially contaminated with the random Gaussian noise and the random harmonic noise is implemented to verify the proposed method. Afterwards, a modal test of a flat plate under the high-temperature condition was undertaken using scanning laser Doppler vibrometry (SLDV). The noise reduction and modal parameter identification were exploited to the measured FRFs. Results show that the reconstructed FRFs retained all of the modal features we concerned about after the noise elimination, and the modal parameters are precisely identified. It demonstrates the superiority and effectiveness of the approach.