Aimed at reducing the redundancy of parameters for the power system stabilizer (PSS) and static var compensator (SVC), this paper proposes a method for coordinated control and optimization based on the neighborhood rough set and improved whale optimization algorithm (NRS-IWOA). The neighborhood rough set (NRS) is first utilized to simplify the redundant parameters of the controller to improve efficiency. Then, the methods of the Sobol sequence initialization population, nonlinear convergence factor, adaptive weight strategy, and random differential mutation strategy are introduced to improve the traditional whale optimization algorithm (WOA) algorithm. Finally, the improved whale optimization algorithm (IWOA) is utilized to optimize the remaining controller parameters. The simulation results show that the optimization parameters were reduced from 12 and 18 to 3 and 4 in the single-machine infinity bus system and dual-machine power system, and the optimization time was reduced by 74.5% and 42.8%, respectively. In addition, the proposed NRS-IWOA method exhibits more significant advantages in optimizing parameters and improving stability than other algorithms.