This article suggests the control of current and speed approach to reduce the torque ripple in BLDC motor. Initially, the renewable energy hybrid power system (REHPS) is composed of a generation system of PV, fuel cell (FC), and the storage system of battery bank. This REHPS uses solar power as their main source of electricity during the day. It uses the fuel cell as a secondary source for maintenance at night or during periods of shaded conditions. The novelty of the proposed method is to achieve torque ripple minimization and to control the speed of the BLDC motor. The speed and error torque of the BLDC motor is optimized by mayfly optimization algorithm (MOA). The MOA provides gain parameters of the fractional order proportional–integral–derivative (FOPID) controller. The advantage of the proposed method is to improve the level of dependability and provide flexibility in solving the system error. The proposed model is implemented in MATLAB/Simulink and experimental setup. The results of the proposed method are compared with the existing research techniques such as particle swarm optimization (PSO) and moth flame algorithm (MFA).