The AIoT, with its artificial intelligence capabilities, can further enhance Device‐to‐Device (D2D) communication. Based on Non‐Orthogonal Multiple Access (NOMA), D2D technology can effectively alleviate wireless spectrum resource pressure and improve the capacity of heterogeneous cellular networks. However, it also introduces significant system interference issues. In this paper, a resource allocation algorithm is proposed for the NOMA‐D2D heterogeneous cellular network, based on a multi‐agent deep reinforcement learning framework. Firstly, the algorithm allocates appropriate channels to D2D clusters. Then, the power allocation factors and D2D transmit power are jointly optimized to suppress the interference and improve the system performance. Simulation results show that both the channel allocation efficiency and the power control performance of the system can be significantly improved.