The optimal power flow (OPF) is a practical problem in a power system with complex characteristics such as a large number of control parameters and also multi-modal and non-convex objective functions with inequality and nonlinear constraints. Thus, tackling the OPF problem is becoming a major priority for power engineers and researchers. Many metaheuristic algorithms with different search strategies have been developed to solve the OPF problem. Although, the majority of them suffer from stagnation, premature convergence, and local optima trapping during the optimization process, which results in producing low solution qualities, especially for real-world problems. This study is devoted to proposing an effective hybridizing of whale optimization algorithm (WOA) and a modified moth-flame optimization algorithm (MFO) named WMFO to solve the OPF problem. In the proposed WMFO, the WOA and the modified MFO cooperate to effectively discover the promising areas and provide high-quality solutions. A randomized boundary handling is used to return the solutions that have violated the permissible boundaries of search space. Moreover, a greedy selection operator is defined to assess the acceptance criteria of new solutions. Ultimately, the performance of the WMFO is scrutinized on single and multi-objective cases of different OPF problems including standard IEEE 14-bus, IEEE 30-bus, IEEE 39-bus, IEEE 57-bus, and IEEE118-bus test systems. The obtained results corroborate that the proposed algorithm outperforms the contender algorithms for solving the OPF problem.