The extraction of photovoltaic (PV) panels from remote sensing images is of great significance for estimating the power generation of solar photovoltaic systems and informing government decisions. The implementation of existing methods often struggles with complex background interference and confusion between the background and the PV panels. As a result, the completeness and edge clarity of PV panel extraction results are compromised. Moreover, most previous studies have overlooked the unique color characteristics of PV panels. To alleviate these deficiencies and limitations, a method for extracting photovoltaic panels from high-resolution optical remote sensing images guided by prior knowledge (PKGPVN) is proposed. Firstly, aiming to address the problems related to missed extractions and background misjudgments, a Photovoltaic Index (PVI) based on visible images in the three-band is constructed to serve as prior knowledge to differentiate between PV panels and non-PV panels. Secondly, in order to strengthen information interaction between shallow features and deep features and enhance the accuracy and integrity of results, a Residual Convolution Hybrid Attention Module (RCHAM) is introduced into the skip-connection of the encoding–decoding structure. Finally, for the purpose of reducing the phenomenon of blurred edges, a multilevel Feature Loss (FL) function is designed to monitor the prediction results at different scales. Comparative experiments are conducted with seven methods, including U-Net, on publicly available datasets. The experimental results show that our PKGPVN achieves superior performance in terms of evaluation metrics such as IoU (above 82%), Precision (above 91%), Recall (above 89%), and F1-score (above 90%) on the AIR-PV dataset. Additionally, the ablation experiments illustrate the effectiveness of our key parts. The proposed method reduces the phenomena of missed extractions and background misjudgments effectively while producing highly accurate results with clear boundaries.