Damage identification based on modal parameters is an important approach in structural health monitoring (SHM). Generally, traditional objective functions used for damage identification minimize the mismatch between measured modal parameters and the parameters obtained from the finite element (FE) model. However, during the optimization process, the repetitive calculation of structural modes is usually time-consuming and inefficient, especially for large-scale structures. In this paper, an improved objective function is proposed based on certain characteristics of the peaks of the frequency response function (FRF). Traditional objective functions contain terms that quantify modal shapes and/or natural frequencies. Here, it is proposed to replace them by the FRF of the FE model, which allows the repeated full modal analysis to be avoided and thus increases the computational efficiency. Moreover, the efficiency is further enhanced by employing the substructural virtual distortion method (SVDM), which allows the frequency response of the FE model of the damaged structure to be quickly computed without the costly re-analysis of the entire damaged structure. Finally, the effectiveness of the proposed method is verified using an eight-story frame structure model under several damage cases. The damage location and extent of each substructure can be identified accurately with 5% white Gaussian noise, and the optimization efficiency is greatly improved compared with the method using a traditional objective function.