Operational modal analysis (OMA) has been utilized to extract structural dynamic characteristics by only using the output responses. In many cases of OMA, the problem of harmonic components, which is caused by periodic excitation components to the structure, may occur, and may lead to erroneous modal identification. In this paper, an optimized harmonic indicator function named the enhanced spectral kurtosis (ESK) is proposed to improve the effectiveness of the harmonic components detection. Moreover, a new algorithm based on virtual excitation assumption is presented to remove the harmonic components in modal parameters estimation. Finally, the quality of the proposed method is compared with that of the conventional method using a numerical simulation and a practical experiment.