Device-to-device (D2D) communication with direct terminal connection is a promising candidate for 5G communication, which increases the capacity of cellular networks and spectral efficiency. Introducing D2D communication to cellular users (CUs) will increase system capacity, and CUs will provide reusable channel resources for D2D users (DUs). However, the sharing of channel resources between CUs and DUs will lead to cofrequency interference and affect the communication quality of user terminals. As a means of improving spectrum utilization and solving cofrequency interference problems, a one-to-many D2D communication system model is established in cellular networks. Through model analysis, the interference between CUs and DUs is correlated with their distance from one another. Considering the different interference of CUs to DUs at different distances, an algorithm for resource allocation based on distance grouping is proposed. With this algorithm, DUs will reuse channel resources of CUs within a reasonable distance in the group, and interference between DUs and CUs will be minimized. The improved particle swarm optimization algorithm is used to solve the optimal power, to achieve the maximum transmission rate of the system. Simulated results show that the algorithm will significantly improve system throughput and performance while also lowering the computational complexity of the algorithm, enabling the whole system to have better communication quality.