The development of the relativistic all-order method where all single, double, and partial triple excitations of the Dirac-Hartree-Fock wave function are included to all orders of perturbation theory led to many important results for study of fundamental symmetries, development of atomic clocks, ultracold atom physics, and others, as well as provided recommended values of many atomic properties critically evaluated for their accuracy for large number of monovalent systems. This approach requires iterative solutions of the linearized coupled-cluster equations leading to convergence issues in some cases where correlation corrections are particularly large or lead to an oscillating pattern. Moreover, these issues also lead to similar problems in the CI+all-order method for many-particle systems. In this work, we have resolved most of the known convergence problems by applying two different convergence stabilizer methods, reduced linear equation (RLE) and direct inversion of iterative subspace (DIIS). Examples are presented for B, Al, Zn + , and Yb + . Solving these convergence problems greatly expands the number of atomic species that can be treated with the all-order methods and is anticipated to facilitate many interesting future applications.