Due to the high penetration of renewable energy sources into the electrical power network, overcurrent relays coordination with highly sensitive and selective protection systems are now two of the most important power protection concerns. In this research, an optimal coordination strategy utilising a new hybrid tripping scheme based on current–voltage characteristics has been devised for overcurrent relays in a power network coupled to a photovoltaic system. This research develops and proves a new optimal coordination scheme based on two optimisation methods, the vibrating particles system and particle swarm optimisation algorithms, in consideration of the impact of renewable sources on fault characteristics. The new optimal coordination approach aims to improve the sensitivity and dependability of the protection system by reducing the tripping time of the overcurrent relays by employing a new hybrid tripping scheme. A specific case study, Conseil International des Grands Réseaux Electriques (CIGRE) distribution network connected to two photovoltaic systems is constructed and presented utilising Industrial software (namely ETAP), and the outcomes of the proposed optimal coordination scheme are compared with standard and recent characteristics from the literature. The hybrid tripping scheme and optimisation techniques are evaluated using different fault and power network model scenarios. The results show that the optimal hybrid tripping scheme provided successfully decreases the overall operating time of the overcurrent relays and increases the sensitivity of the relay during all fault scenarios. The reduction in overall time for the proposed hybrid tripping scheme was 35% compared to the literature for the scenario of a power grid with and without photovoltaic systems.