This paper introduces a new protection system for solar photovoltaic generator (SPVG)-connected networks. The system is a combination of voltage-restrained overcurrent relays (VROCRs) and directional overcurrent relays (DOCRs). The DOCRs are implemented to sense high fault current on the grid side, and VROCRs are deployed to sense low fault current supplied by the SPVG. Furthermore, a novel challenge for the optimal coordination of DOCRs-DOCRs and DOCRs-VROCRs is formulated. Due to the inclusion of additional constraints of VROCR, the relay coordination problem becomes more complicated. To solve this complex problem, a hybrid Harmony Search Algorithm-Bollinger Bands (HSA-BB) method is proposed. Also, the lower and upper bands in BB are dynamically adjusted with the generation number to assist the HSA in the exploration and exploitation stages. The proposed method is implemented on three different SPVG-connected networks. To exhibit the effectiveness of the proposed method, the obtained results are compared with the genetic algorithm (GA), particle swarm optimization (PSO), cuckoo search algorithm (CSA), HSA and hybrid GA-nonlinear programming (GA-NLP) method. Also, the superiority of the proposed method is evaluated using descriptive and nonparametric statistical tests.