In this paper, the physical layer security is studied in device‐to‐device (D2D) communications overlaying cellular network where a D2D pair communicates with each other through an untrusted amplify‐and‐forward (AF) relay. Meanwhile, an untrusted relay acts as an eavesdropper in the network. In addition, a jammer is used to create secure communications between D2D pairs. To obtain ergodic secrecy rate (ESR), two scenarios of without jammer (WoJ) and with jammer (WJ) are investigated. In both scenarios, first, the optimal power allocation (OPA) problem is formulated and solved, and then, new closed‐form solutions are presented by calculating the ESR. Moreover, for the WJ scenario, the slope and power offset of the ESR in high signal‐to‐noise ratio are evaluated. Simulation results are presented to determine the importance of OPA and using a jammer to improve the ESR. In addition, the effects of increasing the distance between D2D pairs with each other and with untrusted relays are investigated. The simulation results validate our closed‐form expressions and show that the ESR improves with OPA, and the ESR of the WJ scenario is higher than the WoJ scenario.

No abstract