The identification of protein-peptide binding sites significantly advances our understanding of their interaction. Recent advancements in deep learning have profoundly transformed the prediction of protein-peptide binding sites. In this work, we describe the Geometric Attention-based networks for Peptide binding Sites identification (GAPS). The GAPS constructs atom representations using geometric feature engineering and employs various attention mechanisms to update pertinent biological features. In addition, the transfer learning strategy is implemented for leveraging the pre-trained protein-protein binding sites information to enhance training of the protein-peptide binding sites recognition, taking into account the similarity of proteins and peptides. Consequently, GAPS demonstrates state-of-the-art (SOTA) performance in this task. Our model also exhibits exceptional performance across several expanded experiments including predicting the apo protein-peptide, the protein-cyclic peptide, and the predicted protein-peptide binding sites. Overall, the GAPS is a powerful, versatile, stable method suitable for diverse binding site predictions.