The location of a logistics center is very important in a logistics system, as the success of the location determines the whole logistics system’s structure, shape, and mode, and not only affects the logistics center’s own operating costs, performance, and future development, but also affects the operation of the entire logistics system. Therefore, the selection of the location for a logistics center has great significance for improving the efficiency of regional logistics and optimizing the structure of a logistics system. This study constructed a multi-factor constrained P-median site-selection model to optimize the locations of logistics centers to improve the efficiency of logistics and optimize the structure of the logistics system in a region. The results show that the optimal distribution of logistics center sites and the coverage of freight capacity demand derived from the particle swarm algorithm are more balanced than those derived by the other algorithm. Following the comparison of the results for the utility of the optimized layout points solved by the particle swarm algorithm and the immune genetic algorithm, it is concluded that the optimal fitness value obtained by the particle swarm algorithm is lower than the other. It is proven that the particle swarm algorithm of the P-median site-selection model under this multi-factor constraint has some reference value for the selection of the sites of multi-logistics centers.