Confronted with the challenges posed by climate change and the ongoing energy transition, solar energy is one of the important new energy sources, and the tower solar power plant has become an innovative solution to promote clean energy development. The optimization of heliostat field layout constitutes a crucial aspect in enhancing the operational efficiency of a concentrated solar power tower plant. Currently, the optimization of heliostat field layout has garnered widespread attention. In this paper, we propose the swarm optimization algorithm with niching and elite competition called NECSO to solve the large-scale heliostat field layout optimization. First, aiming to increase diversity and heterogeneity within the population, we employ a random grouping strategy to partition the population into distinct sub-swarms. Then, we design a niching and elite competition mechanism to harmonize the performance of the exploration. The niching competition is carried out within any sub-swarm to enhance the explorability of particles. The elite competition occurs between the elites which select from each sub-swarm to improve the convergence of particles. Additionally, we develop a mathematical model for the optimization of heliostat field layout. This model employs currently advanced computational methods, facilitating prompt and precise calculation of the optical efficiency in the heliostat field layout. To evaluate the performance of NECSO, we design 15 practical cases of heliostat field layout with varying scales. And then, we conduct comparative experiments with eight currently mainstream and excellent algorithms. The results indicate that NECSO exhibits competitive performance in solving the heliostat field layout optimization, particularly in large-scale cases.