NV centers are among the most promising platforms in the field of quantum sensing. Magnetometry based on NV centers, especially, has achieved concrete development in areas of biomedicine and medical diagnostics. Improving the sensitivity of NV center sensors under wide inhomogeneous broadening and fieldamplitude drift is a crucial issue of continuous concern that relies on the coherent control of NV centers with high average fidelity. Quantum optimal control (QOC) methods provide access to this target; nevertheless, the high time consumption of current methods due to the large number of needful sample points as well as the complexity of the parameter space has hindered their usability. In this paper, we propose the Bayesian estimation phase-modulated (B-PM) method to tackle this problem. In the case of the state transforming of an NV center ensemble, the B-PM method reduced the time consumption by more than 90% compared with the conventional standard Fourier basis (SFB) method while increasing the average fidelity from 0.894 to 0.905. In the AC magnetometry scenario, the optimized control pulse obtained with the B-PM method achieved an eight-fold extension of coherence time T2 compared with the rectangular π pulse. Similar application can be made in other sensing situations. As a general algorithm, the B-PM method can be further extended to the open- and closed-loop optimization of complex systems based on a variety of quantum platforms.