Solving nonlinear equation systems for engineering applications is one of the broadest and most essential numerical studies. Several methods and combinations were developed to solve such problems by either finding their roots mathematically or formalizing such problems as an optimization task to obtain the optimal solution using a predetermined objective function. This paper proposes a new algorithm for solving square and nonsquare nonlinear systems combining the genetic algorithm (GA) and the homotopy analysis method (HAM). First, the GA is applied to find out the solution. If it is realized, the algorithm is terminated at this stage as the target solution is determined. Otherwise, the HAM is initiated based on the GA stage’s computed initial guess and linear operator. Moreover, the GA is utilized to calculate the optimum value of the convergence control parameter (h) algebraically without plotting the h-curves or identifying the valid region. Four test functions are examined in this paper to verify the proposed algorithm’s accuracy and efficiency. The results are compared to the Newton HAM (NHAM) and Newton homotopy differential equation (NHDE). The results corroborated the superiority of the proposed algorithm in solving nonlinear equation systems efficiently.