In recent years, with the growing popularity of smart microgrids in distribution networks, the effective coordination of directional overcurrent relays (DOCRs) has presented a significant challenge for power system operators due to the intricate and nonlinear nature of their optimization model. Hence, this study proposes a hybrid GA-SQP algorithm to enhance the coordination of directional overcurrent relays (DOCRs) in radial and non-radial interconnected distributed power networks. The proposed approach combines the advantages of both the genetic algorithm (GA) and sequential quadratic programming (SQP) methods to optimize the objective function of relay coordination in the best manner. Thus, the proposed hybrid techniques improved the convergence of the problem and increased the likelihood of obtaining a globally optimal solution. Finally, to validate the effectiveness of the proposed algorithm, it was tested through three case studies involving the IEEE 3-Bus, 8-Bus, and modified 30-Bus distribution networks. In addition, the results were compared to those obtained using previous methods. The results obtained from the comparison of the proposed method and recent advanced research indicate that the proposed optimization approach is preeminent in terms of accuracy and total operating time as well as the continuity of the minimum margin time requirements between the primary/backup relay pairs.