The smooth orthogonal decomposition method (SOD) is an efficient algorithm that can be used to extract modal matrix and frequencies of lightly damped vibrating systems. It uses the covariance matrices of output-only displacement and velocity responses to form a generalized eigenvalues problem (EVP). The mode shape vectors are estimated by the eigenvectors of the EVP. It is stated in this work that the accuracy of the SOD method is mainly affected by the correlation characteristic of modal coordinate responses. For the damped vibration systems, biases will be contained in the results of using the SOD. Therefore, an iterative smooth orthogonal decomposition (ISOD) method is proposed to identify modal parameters of the damped system from the covariance matrices of the displacement, velocity, and acceleration responses. The modal matrix given by the SOD method is updated in each iteration step using a transformation matrix. The transformation matrix can be efficiently computed using a set of analytical formulations. Meanwhile, natural frequencies and damping ratios are obtained by using a simple search method. The performance of the proposed ISOD method is verified by numerical and experimental studies. The results demonstrate that, by considering the correlation of modal responses, the ISOD method can be used to extract accurately the modal information of vibration systems with coupled modes.