Accurate position information is crucial for unmanned aerial vehicles (UAV) to execute tasks. To balance the contradiction between the payload and localization accuracy of rotary UAVs, a cooperative localization method for UAV swarms based on the factor graph model is studied in this paper. Each UAV is equipped with a local factor graph model. A distributed factor graph model-solving method, AGA-Gauss–Newton conjugate gradient (GNCG), which combines an adaptive genetic algorithm and an improved GNCG algorithm, is proposed. The issue of falling into local optimal solutions was addressed by configuring the crossover and mutation behaviors of the genetic algorithm into an adaptive mode. The Gauss–Newton algorithm (GNQR) was improved using a conjugate gradient iteration, which effectively reduced the operation time of the algorithm. The simulation results indicate that the AGA-GNCG algorithm improves the localization accuracy with respect to the East–North–Up (ENU) frame by 58.8%, 60.6%, and 57.4% relative to the GNQR. Moreover, as the number of UAVs increases, the improved GNCG algorithm exhibits a significant improvement in computational efficiency compared to that of the GNQR algorithm.