Source mask optimization (SMO) is an effective method for improving the image quality of high-node lithography. Reasonable algorithm optimization is the critical issue in SMO. A GA-APSO hybrid algorithm, combining genetic algorithm (GA) and adaptive particle swarm optimization (APSO), was proposed to inversely obtain the global optimal distribution of the pixelated source and mask in the lithographic imaging process. The computational efficiency was improved by combining the GA and PSO algorithms. Additionally, the global search and local search were balanced through adaptive strategies, leading to a closer result to the global optimal solution. To verify the performance of GA-APSO, simple symmetric patterns and complex patterns were optimized and compared with GA and APSO, respectively. The results show that the pattern errors (PEs) of the resist image optimized by GA-APSO were reduced by 40.13–52.94% and 10.28–33.31% compared to GA and APSO, respectively. The time cost of GA-APSO was reduced by 75.91–87.00% and 48.43–58.66% compared to GA and APSO, respectively. Moreover, repeated calculation showed that the GA-APSO results were relatively stable. The results demonstrate the superior performance of GA-APSO in efficiency, accuracy, and repeatability for source and mask optimization.