Wireless mesh networks are being considered as the most adequate topology for deployment in the neighborhood area network (NAN) domain in the smart‐grid infrastructure, because its features such as self‐organizing, scalability, and cost‐efficiency complement to the NAN requirements. To provide security functionality to the NAN, the key refreshment strategy of the simultaneous authentication of equals or the efficient mesh security association protocol is an efficient way to make the network more resilient against various cyberattacks. However, it is discovered that when the key refreshment strategy is used, the efficient mesh security association protocol demonstrates a security vulnerability, leading to denial of service attacks. In this paper, a simple hash‐based encryption scheme is proposed to prevent the unprotected messages from being replayed by the adversary with an enhancement to the key refreshment scheme to improve the resilience of the mesh key holder security handshake. The Protocol Composition Logic is used to describe the logical correctness of the proposed scheme, while the Process Analysis Toolkit is used to formally verify the security functionality against the malicious attacks. The efficiency analysis and the simulation results prove that the proposed scheme is reliable and efficient. Copyright © 2016 John Wiley & Sons, Ltd.