Internet of Things (IoT) is a developing technology in our time that is prone to security problems as it uses wireless and shared networks. A challenging scenario in IoT environments is Device-to-Device (D2D) communication that an authentication server as a trusted third-party, does not involve in the authentication and key agreement process. It is only involved in the process of allocating long-term secret keys and their update. A lot of authentication protocols have been suggested for such situations. This article demonstrated that three state-of-the-art related protocols failed to remain anonymous, insecure against key compromise impersonation (KCI) attack, and clogging attack. To counter the pitfalls of them, a new D2D mutual authentication and key agreement protocol is designed here. The proposed protocol is anonymous, untraceable, and highly secure. Moreover, there is no need for a secure channel to generate a pair of private and public keys in the registration phase.) Formal security proof and security analysis using BAN logic, Real-Or-Random (ROR) model, and Scyther tool showed that our proposed protocol satisfied security requirements. Furthermore, communication cost, computation cost, and energy consumption comparisons denoted our schema has better performance, compared to other protocols.