In this paper, the multiple-rendezvous trajectory optimization problem is studied, which refers to optimizing rendezvous epochs for a single spacecraft to service multiple targets. It is found that the problem has multiple properties, such as a predictable trough number in each dimension. A novel Lambert property-based swarm search algorithm (LPSSA) that incorporates this problem-specific knowledge is proposed to improve efficiency. The design aims to improve the global search capability in the prophase and the local search capability in the anaphase. The algorithm has two different branches and is switched by a proposed complexity index. When the index is relatively large, the initial variables are distributed in some priority trough regions, and the update mechanism is divided into three stages. When the index is relatively small, the initial variables are first distributed to all trough regions and then selected. Multiple subpopulations search independently in the iterative process, and an annexation mechanism is introduced. An improved method for handling boundary conditions is applied in both branches. Besides, the improved mechanisms can also be combined with other swarm search algorithms. Numerical simulations show that the proposed method has a more stable convergence performance and a more optimal solution than the conventional algorithms.