To achieve high measurement accuracy without the restriction of fringe number, a high-accuracy phase shifting retrieval approach based on phase shifts search (PSS) is proposed. Firstly, by setting the search conditions related to the phase shifts and using the PSS method to obtain the phase shifts between the phase shifting interferograms, subsequently, the phase distribution of the sample can be accurately obtained by using the obtained phase shifts. To our knowledge, the proposed method has the highest accuracy among the multi-step arbitrary phase shifting algorithms, and it is not limited to the number of fringes and the distribution of phase shifts between the interferograms. In the case of sparse fringes, the phase can still be retrieved with high accuracy, which proves that the proposed method is a high accuracy method with fewer constraints and wider applications. The feasibility and superiority of this method are proved by the simulation analysis and experiment.