In this paper, Power System Stabilizer (PSS) based on hybrid Sliding Mode Controller (SMC) and fractional order PID controller (pIAn 11) is proposed for optimal control of power system, using a new metaheuristic optimization Bat algorithm (BA) inspired by the echolocation behavior to improve power system stability. The aim of this study is to find robust controller for demonstrating the availability of the proposed controller, and to achieve best desired output. Here, we have chosen SMC as one of the most effective control methodologies for adjusting the states of the system to their preferred values to supply the excellent damping under severe disturbances. The problem of SMC-FOPID design is transformed to an optimization problem based on performance index, which is Integral of the Time Weighted Absolute Error (lTAE). Where, BA has been employed to adjust the optimal controller parameters. The effectiveness of SMC-FOPID has been tested on a Single Machine Infinite Bus (SMIB) power system under different cases, including operating. The performance of SMC-FOPID controller in power system is compared with a conventional PSS. The simulation results greatly indicate the validity and effectiveness of the proposed controller, and superior robust performance for improvement power system stability compared with the PSS controller for each case.