The cryptographic quality of the cipher algorithm depends on its key security. Typically, cryptographic applications provide unpredictable keys to strengthen the security of the plaintext. Despite advancements made in the field of cryptography, security loopholes still exist. As such, this paper proposed a key security of Playfair cipher algorithm by applying Playfair cipher 16x16 matrix, XOR operator, two's complement, and bit swapping. It also describes randomness tests on the proposed method that has been implemented on the existing application. We selected seven suitable randomness tests provided by NIST Test Suite. Experimental results show that the generated binary sequences are random as the proposed method passed all selected randomness tests which include frequency (Monobit) test, frequency test within a block, run test, test for the longest run of ones in a block, discrete Fourier transform, approximate entropy test, and cumulative sums test. In fact, among the different length of keys (10, 20, 30, and 40), we obtained P-values ranging from 0.01 to 1.00. It only means that no pattern has found, thus the key is unpredictable.