Transformer-Based Deep Learning Model with Latent Space Regularization for CRISPR-Cas Protein Sequence Classification

Abstract

The discovery of the CRISPR-Cas system has significantly advanced genome editing, offering vast applications in medical treatments and life sciences research. Despite their immense potential, the existing CRISPR-Cas proteins still face challenges concerning size, delivery efficiency, and cleavage specificity. Addressing these challenges necessitates a deeper understanding of CRISPR-Cas proteins to enhance the design and discovery of novel Cas proteins for precision gene editing. In this study, we performed extensive deep-learning research on CRISPR-Cas proteins, aiming to develop a classification model capable of distinguishing CAS from non-CAS proteins, as well as discriminating sub-categories of CAS proteins, specifically CAS9 and CAS12. We developed two types of deep learning models: 1) a transformer encoder-based classification model, trained from scratch; and 2) a large protein language model fine-tuned on ProtBert, pre-trained on more than 200 million proteins. To boost learning efficiency for the model trained from scratch, we introduced a novel margin-based loss function to maximize inter-class separability and intra-class compactness in protein sequence embedding latent space of a transformer encoder. The experimental results show that the Fine-Tuned ProtBert-based (FTPB) classification model achieved accuracies of 99.06\%, 94.42\%, 96.80\%, 97.57\% for CAS9 vs. Non-CAS, CAS12 vs. Non-CAS, CAS9 vs. CAS12, and multi-class classification of CAS9 vs. CAS12 vs. Non-CAS, respectively. The Latent Space Regularized Max-Margin Transformer (LSRMT) model achieved classification accuracies of 99.81\%, 99.81\%, 99.06\%, 99.27\% for the same tasks, respectively. These results demonstrate the effectiveness of the proposed Max-Margin-based latent space regularization in enhancing model robustness and generalization capabilities. Remarkably, the LSRMT model, even when trained on a significantly smaller dataset, outperformed the fine-tuned state-of-the-art large protein model. The high classification accuracies achieved by the LSRMT model demonstrate its proficiency in identifying discriminative features of CAS proteins, marking a significant step towards advancing our understanding of CAS protein structures in future research endeavors.