In radio‐frequency identification (RFID) authentication technology, the authentication schemes between reader and tag based on public‐key cryptography (PKC) are much better than those based on symmetric‐key cryptography in terms of expanding the scale of RFID applications and the style of providing service, while the limitation of resource consumption and computation capability of RFID tags makes it difficult to apply traditional PKC to RFID authentications. The cryptoGPS protocol based on PKC proposed by Mcloone and Robshaw suits low‐cost RFID system well, but it just achieves one‐way authentication, and the authentication times are very limited, which makes it vulnerable to coupons‐exhausted DoS attacks. To solve these problems, cryptoGPS protocol is greatly improved to realize the mutual authentication between RFID reader and the tag. In the improved protocol, a readers' public key distribution scheme is proposed to support the mutual authentication and a coupons updating algorithm is presented to resist the aforementioned DoS attack. Moreover, a modified Rabin encryption algorithm and a parameter generation method based on Low Hamming Weight technology are proposed to optimize the implementation of the proposed authentication protocol. And a feasible hardware structure of the protocol is also given. The protocol's simulation results show that the scheme just needs 3232 equivalent gates, and the maximum time of single step is 3.3 ms (500 k clock). The scheme is suitable for the low‐cost tags. Copyright © 2014 John Wiley & Sons, Ltd.