To minimize the major decline in direction of arrival (DOA) estimation performance for an acoustic vector sensor array (AVSA) with the coexistence of axial deviation and non-uniform noise, a two-step iterative minimization (TSIM) method is proposed in this paper. Initially, the axial deviation measurement model of an AVSA is formulated by incorporating the disturbance parameter into the signal model, and then a novel AVSA manifold matrix is defined to estimate the sparse signal power and noise power mutually. After that, to mitigate a joint optimization problem to achieve the sparse signal power, the noise power and the axial deviation matrix, two auxiliary cost functions, are presented based on the covariance matrix fitting (CMF) criterion and the weighted least squares (WLS), respectively. Furthermore, their analytical expressions are also derived. In addition, to further enhance their prediction accuracy, the estimated axial deviation matrix is modified based on its specific structural properties. The simulation results demonstrate the superiority and robustness of the proposed technique over several conventional algorithms.