In this paper, an efficient mask optimization method is proposed based on the improved genetic algorithm (GA), which can obviously enhance the performance of digital micromirror device (DMD) maskless lithography. Through meticulous exploration and comparison of multiple optimization techniques, the simulated annealing improved genetic algorithm (SA-GA) method was identified as the most effective improved algorithm, which notably improves the lithography simulation outcomes to achieve mask optimization objectives. Results demonstrated the effectiveness of this method, with the improvement percentage improved by up to 88% and to 75% for circular and heart-shaped images compared to traditional Hopkins lithography simulation. The remarkable effect of improved GA in enhancing the quality of DMD digital lithography shows that it will have great potential in micro-fabrication applications, and paves the way for the realization of high-fidelity and efficient DMD digital lithography technology, which has star versatility and adaptability in the field of microelectronics manufacturing.