The Wahba problem is the task of constrained optimization seeking the matrix from SO(3), which maximally converges (based on the least squares criterion) two sequences of unit vectors. The solution of this task is vital for satellite attitude determination using star trackers. An iterative method for solving the Wahba problem is proposed. Each iteration of the proposed method is reduced to sequential rotation of the vectors and solving the system of linear algebraic equations. Usage of the method implies that the corresponding vectors of both sequences are located sufficiently close to each other. Two variants of the method are proposed, having linear and quadratic convergence. The Wahba problem solution is interpreted in terms of finding the angular velocity of a system of material points, which have certain angular momentum. Taking into consideration the characteristics of state-of-the-art star trackers, one to two iterations are sufficient for finding the optimal solution using the small-angle rotation method. The primary advantage of the proposed method as compared with classical methods based on calculation of eigenvectors and singular decomposition is the simplicity of its implementation. © The Authors.Published by SPIE under a Creative Commons Attribution 3.0 Unported License. Distribution or reproduction of this work in whole or in part requires full attribution of the original publication, including its DOI.