The transition from traditional to renewable energy sources is a critical issue in current energy-generation systems, which aims to address climate change and the increased demand for energy. This shift, however, imposes additional burdens on control systems to maintain power system stability and quality within predefined limits. Addressing these challenges, this paper proposes an innovative Modified Hybrid Fractional-Order (MHFO) automatic voltage regulator (AVR) equipped with a fractional-order tilt integral and proportional derivative with a filter plus a second-order derivative with a filter FOTI-PDND2N2 controller. This advanced controller combines the benefits of a (FOTI) controller, known for enhancing dynamic performance and steady-state response, with a (PDND2N2) controller to improve system robustness and adaptability. The proposed MHFO controller stands out with its nine tunable parameters, providing more extensive control options than the conventional three-parameter PID controller and the five-parameter FOPID controller. Furthermore, a recent optimization approach using a growth optimizer (GO) has been formulated and applied to optimally adjust the MHFO controller’s parameters simultaneously. The performance of the proposed AVR based on the MHFO-GO controller is scrutinized by contrasting it with various established and developed optimization algorithms. The comparative study shows that the AVR based on the MHFO-GO controller surpasses other AVR controllers from the stability, robustness, and dynamic response speed points of view.