In order to obtain accurate nano-film characteristic parameters in the ellipsometry measurement process, an optimization algorithm for solving the thickness and complex refractive index of nano films by spectroscopic ellipsometry is proposed. An improved adaptive genetic algorithm (IAGA) has been proposed to process nano-film data, this method combines the evolutionary algebraic attenuation factor with the adaptive genetic algorithm. It can solve the problem that the genetic algorithm is premature and easy to fall into the local optimization. The algorithm is used to calculate the film parameters of silicon dioxide nano film thickness standard template with standard value of 49.7±0.4 nm in this paper. The results show that the relative error of the calculation results of the film thickness is less than 3%, and the error of refractive index is less than 0.1. At the same time, it is verified by experiments that the IAGA algorithm model can effectively optimize the number of iterations, and has the advantages of fast convergence speed and high measurement efficiency.