Electronic commerce(E-commerce) transactions require secure communication to protect sensitive information such as credit card numbers, personal identification, and financial data from unauthorized access and fraud. Encryption using public key cryptography is essential to ensure secure electronic commerce transactions. RSA and Rabin cryptosystem algorithms are widely used public key cryptography techniques, and their security is based on the assumption that it is computationally infeasible to factorize the product of two large prime numbers into its constituent primes. However, existing variants of RSA and Rabin cryptosystems suffer from issues like high computational complexity, low speed, and vulnerability to factorization attacks. To overcome the issue, this article proposes a new method that introduces the concept of fake-modulus during encryption. The proposed method aims to increase the security of the Rabin cryptosystem by introducing a fake-modulus during encryption, which is used to confuse attackers who attempt to factorize the public key. The fake-modulus is added to the original modulus during encryption, and the attacker is unable to distinguish between the two. As a result, the attacker is unable to factorize the public key and cannot access the sensitive information transmitted during electronic commerce transactions. The proposed method’s performance is evaluated using qualitative and quantitative measures. Qualitative measures such as visual analysis and histogram analysis are used to evaluate the proposed system’s quality. To quantify the performance of the proposed method, the entropy of a number of occurrences for the pixels of cipher text and differential analysis of plaintext and cipher text is used. When the proposed method’s complexity is compared to a recent variant of the Rabin cryptosystem, it can be seen that it is more complex to break the proposed method—represented as O(ɲ× τ) which is higher than Rabin-P (O(ɲ)) algorithms.