In this paper, a modified incremental harmonic balance (IHB) method combined with Tikhonov regularization has been proposed to achieve the semi-analytical solution for the periodic nonlinear system. To the best of our knowledge, the convergence of the traditional IHB method is bound up with the iterative initial values of harmonic coefficients, especially near the bifurcation point. To this end, the Tikhonov regularization is introduced into the linear incremental equation to tackle the ill-posed situation in the iteration. To this end, the convergence performance of the traditional IHB method has been improved significantly. Moreover, convergence proof of the proposed method also has been given in this paper. Finally, a van der Pol–Duffing oscillator with external excitation and a cubic nonlinear airfoil system with the external store are adopted as numerical examples to illustrate the efficiency and the performance of the presented modified IHB method. The numerical examples show that the results achieved by the proposed method are in excellent agreement with the Runge–Kutta method, and the accuracy is not significantly reduced compared with the traditional IHB method. Especially, the modified IHB method also can converge to the exact solution from the initial values that the traditional IHB method cannot converge in both examples.