Randomizing the layout of the keypad has been proposed to improve the security of PIN entry. However, there has been no empirical quantification of its impact on usability and security. We present the first usability (N=17) and security (N=24) evaluations to compare PIN entry with the standard vs randomized layout. Our results show that randomizing the layout increases resistance to shoulder surfing and thermal attacks significantly, and has a very minor impact on entry accuracy, but it increases entry time (from ≈1.4 seconds to ≈2 seconds). We discuss how this simple approach can improve security with little impact on usability. CCS CONCEPTS• Human-centered computing → Interaction techniques; • Security and privacy → Authentication.