Overlay measurement error is a critical technical issue in the production of highly stacked semiconductor devices, including VNAND memory chips, CMOS image sensors, and three-dimensional packages. Complicated device structure and stacked structure increase the measurement error of overlay alignment mark position. Inverse optical design or system optimization is required to improve overlay metrology accuracy and measurement robustness. Illumination light source wavelength, source bandwidth, illumination mode, and imaging pupil filter can be optimized for overlay metrology signal with various kinds of complicated device structure. We proposed a practical inverse optical solution to improve the accuracy of overlay metrology. The inverse optical design consists of overlay mark reflectance estimation and optical configuration optimization. Both the estimation and optimization are accelerated using a quasi-two-dimensional partially coherent imaging model. We achieve more than 50 times faster imaging simulation acceleration compared to a conventional simulation algorithm for partially coherent illumination imaging with practical accuracy. Further improvement can be realized with an introduce of an overlay mark reflectance matrix computed by a rigorous electromagnetic analysis simulation for each specific device structure. This robust and practical inverse solution can help improve the overlay accuracy and manufacturing yield of highly complex three-dimensional devices.