The ever-growing threats in cybersecurity growing with the rapid development of quantum computing, necessitates the development of robust and quantum-resistant cryptographic systems. This paper introduces a novel cryptosystem, Public Key Cryptosystem based on Systematic Polar Encoding (PKC-SPE), based on the combination of systematic polar encoding and public-key cryptographic principles. The Systematic Polar Encoding (SPE), derived from the well-established field of polar codes, serves as the foundation for this proposed cryptographic scheme. Here, we have used MATLAB Software to introduce and implement the PKC-SPE Cryptosystem. The paper examines key generation, encryption, and decryption algorithms, providing insights into the adaptability and efficiency of systematic polar encoding in public-key cryptography. We assess the efficiency of the PKC-SPE Cryptosystem in three aspects: key size, computational complexity, and system implementation timings. In addition, we compare the PKC-SPE Cryptosystem with PKC-PC cryptosystem and find that it has reduced key sizes ($$P_{r}$$
P
r
= 0.8436 kbytes). The results obtained through simulations validate the effectiveness of the proposed cryptosystem and highlighting its potential for integration into real-world communication systems. Thus, in the paradigm shift to quantum computing, the PKC-SPE cryptosystem emerges as a promising candidate to secure digital communication in the quantum computing era.