In order to effectively solve the increasingly prominent network security problems, cryptographic algorithm is the key factor affecting the effectiveness of IPSec VPN encryption. Therefore, this paper mainly studies cryptographic algorithms and puts forward the following solutions: briefly analyze the concept and function of IPSec VPN, as well as the basic theoretical knowledge of IPSec Security Protocol and cryptography, and analyze the traditional cryptography, modern cryptography, symmetric cryptographic algorithms and asymmetric algorithms, and their security. At the same time, the executable and security performances of AES and DES algorithms are compared and analyzed. This paper studies the elliptic curve encryption algorithm ECC, expounds the mathematical basis of realizing the algorithm, and compares and analyzes the security performance and execution efficiency of ECC. Based on the above two algorithms, a hybrid encryption algorithm is proposed, and the realization mechanism of the hybrid encryption algorithm is studied and discussed. The hybrid encryption algorithm combines the advantages of ECC and AES. The algorithm selects 128-bit AES and 256-bit ECC. In order to better cover up plaintext C, AES is used to encrypt information. While enhancing security, speed is also considered. The improved encryption, decryption, and signature authentication algorithms are relatively safe and fast schemes. ECC algorithm is improved, and on this basis, ECC algorithm and AES algorithm are combined. Moreover, HMAC message authentication algorithm is added, and the performance of the improved algorithm is significantly improved.