Given the intricate nature of contemporary energy systems, addressing the control and stability analysis of these systems necessitates the consideration of highly large-scale models. Transient stability analysis stands as a crucial challenge in enhancing energy system efficiency. Power System Stabilizers (PSSs), integrated within excitation control for synchronous generators, offer a cost-effective means to bolster power systems’ stability and reliability. In this study, we propose an enhanced nonlinear control strategy based on synergetic control theory for PSSs. This strategy aims to mitigate electromechanical oscillations and rectify the limitations associated with linear approximations within large-scale energy systems that incorporate thyristor-controlled series capacitors (TCSCs). To dynamically adjust the coefficients of the nonlinear controller, we employ the Fractional Order Fish Migration Optimization (FOFMO) algorithm, rooted in fractional calculus (FC) theory. The FOFMO algorithm adapts by updating position and velocity within fractional-order structures. To assess the effectiveness of the improved controller, comprehensive numerical simulations are conducted. Initially, we examine its performance in a single machine connected to the infinite bus (SMIB) power system under various fault conditions. Subsequently, we extend the application of the proposed nonlinear stabilizer to a two-area, four-machine power system. Our numerical results reveal highly promising advancements in both control accuracy and the dynamic characteristics of controlled power systems.