Peptides have recently emerged as therapeutic molecules against various diseases, and the secondary structure of peptides is a crucial determinant of their bioactivity. However, accurately predicting peptide secondary structures remains a challenging task due to the lack of peptide sequence data and low prediction efficiency caused by limitations in feature engineering. Therefore, we developed PHAT, a deep learning framework based on a hypergraph multi-head attention network and transfer learning for the prediction of peptide secondary structures. Comparative results demonstrated the outstanding performance and robustness of PHAT. In particular, PHAT automatically learns a set of biologically meaningful knowledge on secondary sub-structures, overcoming the limitations of "black-box" in deep learning-based models and providing good interpretability. Additionally, we demonstrated that the structure information derived from PHAT significantly improved the performance of downstream tasks such as the prediction of peptide toxicity and protein-peptide binding sites. Importantly, we further explored the applicability of PHAT for contact map prediction, which can aid in the reconstruction of peptide 3-D structures, thus highlighting the versatility of our model. To facilitate the use of PHAT, we establish an online server which is accessible via https://server.wei-group.net/PHAT/. We expect our work to assist in the design of functional peptides and contribute to the advancement of structural biology research.