In this paper, an efficient mask optimization method for enhanced digital micromirror device lithography quality based on improved particle swarm optimization (PSO) is proposed, which greatly improves the quality of lithography. First, the traditional PSO algorithm is improved by introducing adaptive parameter adjustment to enhance its search ability in complex problems. In addition, in order to avoid premature convergence of the algorithm, a simulated annealing operation is introduced to make it accept the different solution with a certain probability and jump out of the local optimal better. The numerical simulation experiment results showed that the pattern errors between the print image and target pattern were reduced by 93.5%, 95.8%, and 95.6%, respectively. Compared with traditional optimization methods, the proposed algorithm significantly improves the image quality, especially in the aspects of edge contour and pattern fidelity.