The whale optimization algorithm (WOA) is inspired by humpback whale social behavior and WOA is a popular swarm intelligence algorithm. Yet, the WOA does have certain flaws, such as a restricted global search capability and an inconsistent convergence speed, and when dealing with complex optimization problems, WOA is easy to fall into local optimum. To address the WOA’s shortcomings, a modified whale optimization algorithm with convergence and exploitability enhancement called MWOA-CEE is proposed. Three operators, opposition-based learning, exponentially decreasing function, and elite-guided Cauchy mutation, are integrated into the WOA to enhance the exploration performance and avoid local optimum. First, the opposition-based learning used the opposition-based concept to generate the opposite position of the candidate solution, which can enhance the convergence ability and quality of the solution. Then, the exponentially decreasing function balances the exploitation and exploration phases and boosts the exploitation capability. In the end, the elite-guided Cauchy mutation generated a wide disturbance near the current candidate solution, which improves solutions diversity and increases the algorithm’s global search capability. MWOA-CEE is validated on 19 optimization benchmark functions and 14 CEC2014 benchmark functions and used to identify IIR systems. It is compared to seven of the commonly used algorithms. The results prove that MWOA-CEE outperforms the other algorithms in terms of final solution quality and convergence rate.