Antibody represents a specific class of proteins produced by the adaptive immunity as a response to invading pathogens, and mining the information implied in antibody amino acid sequences can benefit both antibody property prediction and novel therapeutic development. Protein-specific pre-training models have been used to extract latent representations from protein sequences containing structural, functional, and homologous information. However, there is still room for improvement in pre-training models on antibody sequence. On the one hand, existing protein pre-training models mainly utilize pre-training language models without fully considering the differences between protein sequences and language sequences; on the other hand, in comparison with other proteins, antibodies possess their uniqueness, which should be incorporated using specifically designed training methods. Here, we present a pre-trained model of antibody sequences, Pre-training with A Rational Approach for antibodies (PARA), that employs a training strategy conforming to antibody sequences patterns and an advanced NLP self-encoding model structure. We show PARA's performance on several tasks by comparing it to several published pre-trained models of antibodies. The results show that PARA significantly outperforms selected antibody pre-training models on these tasks, suggesting that PARA has an advantage in capturing antibody sequence information. To the best of our knowledge, PARA is the first antibody language model that takes into account the features of antibody sequences . We believe that the antibody latent representation provided by PARA can substantially facilitate the studies in relevant areas, such as antibody structure prediction, affinity prediction, and antibody de novo desig