Integration of renewable energy sources (RES) to the grid in today’s electrical system is being encouraged to meet the increase in demand of electrical power and also overcome the environmental related problems by reducing the usage of fossil fuels. Power Quality (PQ) is a critical problem that could have an effect on utilities and consumers. PQ issues in the modern electric power system were turned on by a linkage of RES, smart grid technologies and widespread usage of power electronics equipment. Unified Power Quality Conditioner (UPQC) is widely employed for solving issues with the distribution grid caused by anomalous voltage, current, or frequency. To enhance UPQC performance, Fractional Order Proportional Integral Derivative (FOPID) is developed; nevertheless, a number of tuning parameters restricts its performance. The best solution for the FOPID controller problem is found by using a Coati Optimization Algorithm (COA) and Osprey Optimization Algorithm (OOA) are combined to make a hybrid optimization CO-OA algorithm approach to mitigate these problems. This paper proposes an improved FOPID controller to reduce PQ problems while taking load power into account. In the suggested model, a RES is connected to the grid system to supply the necessary load demand during the PQ problems period. Through the use of an enhanced FOPID controller, both current and voltage PQ concerns are separately modified. The pulse signal of UPQC was done using the optimal controller, which analyzes the error value of reference value and actual value to generate pulses. The integrated design mitigates PQ issues in a system at non-linear load and linear load conditions. The proposed model provides THD of 12.15% and 0.82% at the sag period, 10.18% and 0.48% at the swell period, and 10.07% and 1.01% at the interruption period of non-linear load condition. A comparison between the FOPID controller and the traditional PI controller was additionally taken. The results showed that the recommended improved FOPID controller for UPQC has been successful in reducing the PQ challenges in the grid-connected RESs system.