Dispersion engineering is always the important topic in the field of artificial periodic structures. In particular, topology optimization of composite structures with expected bandgaps plays a key role. However, most reported studies focused on topology optimization for bulk waves, and the optimization for surface wave bandgaps (SWBGs) is still missing. In this paper, we develop a topology optimization framework based on the genetic algorithm and finite element method to design periodic barriers embedded in semi-infinite space for reducing surface waves on demand. The objective functions for SWBGs are proposed based on the energy distribution properties of surface waves. The numerical results show that the optimization framework has stable convergence for this problem and is effective to optimize SWBGs. Considering large SWBGs and low filling fraction of solids, we investigate single-and multi-objective optimizations, respectively, and obtain novel wave barriers with good performance. The beneficial configuration features and mechanism of broadband SWBGs from the optimized results are explored. The results indicate that for the first-order SWBG, most optimized structures consist of a main scatterer at the center and some subsidiary scatterers near the surface. The rigid body resonance of the main scatterer determines the lower edge of SWBG, and the subsidiary scatterers can regulate the upper edge. Higher-order SWBGs are generated from the interaction of multiple scatterers, whose relative distance has a great influence on the position of SWBGs. The optimized structures can make the surface waves propagate far away from the surface within the frequencies of bandgaps, leading to a strong attenuation of surface vibration. In practice, our topological optimization framework is promising in designing high-performance surface wave devices and novel isolating structures for earthquake or environmental vibration in civil engineering.