The performance of smart antenna greatly relies on the efficient use of direction-of-arrival (DOA) estimation techniques for both coherent and non-coherent signals. In practice, DOA estimation problems and difficulties increase when the signals in multipath propagation environments are highly correlated or coherent. Therefore exploring an algorithm which is capable of estimating coherent signals is of great importance. To overcome this problem a new high-resolution modified virtual singular value decomposition (MVSVD) algorithm for DOA estimation of coherent signals is proposed. It is based on the hybrid combination of the virtual array extension singular value decomposition (SVD), and modified MUSIC algorithms. The proposed algorithm provides many features such as: decorrelation of the coherence between the signals without reducing the rank of the covariance matrix or losing the array aperture size; high-resolution and more stability especially at low SNR values; and an increase in the maximal number of detectable signals to M − 1, where M is the number of antenna elements.