The nano-scale spatial positioning of nanoparticles in tumor cells can be achieved through the double-helix point spread functions (DH-PSF). Nevertheless, certain issues such as low light intensity concentration of the main lobes, the influence of the side lobes, and the aberrations of the imaging system result in poor image quality and reduce the positioning accuracy of the fluorescent nanoparticles. In this paper, an iterative optimization algorithm that combines Laguerre–Gaussian modes and Zernike polynomials is proposed. The double-helix point spread function, constructed by the linear superposition of the Laguerre–Gaussian mode and Zernike polynomials, is used to express aberrations in the imaging system. The simulation results indicated that the light intensity concentration of the main lobes is increased by 45.51% upon the use of the optimization process. Based on the simulation results, the phase modulation plate was designed and processed while a 4f positioning imaging system was built. Human osteosarcoma cells, labeled by CdTe/CdS/ZnS quantum dots, were used as samples, and the position imaging experiment was carried out. The image information entropy was used as the clarity evaluation index. The experimental results showed that the image information entropy of the DH-PSF position imaging was reduced from 4.22 before optimization to 2.65 after optimization, and the image clarity was significantly improved. This result verified the effectiveness of the optimization method that was proposed in this work.