Bidirectional asymptotic structure methods have long been used to solve topological optimization problems, but are prone to being stuck in local optimal solutions. To solve this problem, this paper proposed a topology optimization method based on the Bi-directional Evolutionary structure Structural Optimization and Simulated Annealing algorithm (SA-BESO). First, the structural elements of the structural partition are encoded by a dual encoding, where elements are assigned with density values and binary strings. Second, binary strings are crossed and mutated, while criteria for adding and removing structural units are formulated. Then, structures are updated randomly. Finally, the structural compliance of the current structure is evaluated. If the structural compliance of the original structure increases, it will be accepted with a certain probability, thus jumping out of the local optimal solution. Related examples show that the SA-BESO method improves the smoothness of the optimization process and can obtain optimized structures with lower structural compliance and computational cost.