At present, the integration of microgrids into power systems presents significant power quality challenges in terms of the rising adoption of nonlinear loads and electric vehicles. Ensuring the stability and efficiency of the electrical network in this evolving landscape is crucial. This paper explores the implementation of cascading Proportional–Integral (PI-PI) and cascading Fractional-Order PI (FOPI-FOPI) controllers for a Distribution Static Compensator (DSTATCOM) in hybrid microgrids that include photovoltaic (PV) systems and fuel cells. A novel hybrid optimization algorithm, WSO-WOA, is introduced to enhance power quality. This algorithm leverages the strengths of the White Shark Optimization (WSO) algorithm and the Whale Optimization Algorithm (WOA), with WSO generating new candidate solutions and WOA exploring alternative search areas when WSO does not converge on optimal results. The proposed approach was rigorously tested through multiple case studies and compared with established metaheuristic algorithms. The findings demonstrate that the WSO-WOA hybrid algorithm significantly outperforms others in optimizing the PI-PI and FOPI-FOPI controllers. The WSO-WOA algorithm showed an improvement in accuracy, surpassing the other algorithms by approximately 7.29% to 14.1% in the tuning of the PI-PI controller and about 8.5% to 21.2% in the tuning of the FOPI-FOPI controller. Additionally, the results confirm the superior performance of the FOPI-FOPI controller over the PI-PI controller in enhancing the effectiveness of the DSTATCOM across various scenarios. The FOPI-FOPI provided controller a reduced settling time by at least 30.5–56.1%, resulting in marked improvements in voltage regulation and overall power quality within the microgrid.