Recent experimental measurements, without any theoretical guidance, showed that isotropic polarization response can be achieved by increasing the number of QD layers in a QD stack. Here we analyse the polarization response of multi-layer quantum dot stacks containing up to nine quantum dot layers by linearly polarized PL measurements and by carrying out a systematic set of multi-million atom simulations. The atomistic modeling and simulations allow us to include correct symmetry properties in the calculations of the optical spectra: a factor critical to explain the experimental evidence. The values of the degree of polarization (DOP) calculated from our model follows the trends of the experimental data. We also present detailed physical insight by examining strain profiles, band edges diagrams and wave function plots. Multi-directional PL measurements and calculations of the DOP reveal a unique property of InAs quantum dot stacks that the TE response is anisotropic in the plane of the stacks. Therefore a single value of the DOP is not sufficient to fully characterize the polarization response. We explain this anisotropy of the TE-modes by orientation of hole wave functions along the [110] direction. Our results provide a new insight that isotropic polarization response measured in the experimental PL spectra is due to two factors: (i) TM001-mode contributions increase due to enhanced intermixing of HH and LH bands, and (ii) TE110-mode contributions reduce significantly due to hole wave function alignment along the [110] direction. We also present optical spectra for various geometry configurations of quantum dot stacks to provide a guide to experimentalists for the design of multi-layer QD stacks for optical devices. Our results predict that the QD stacks with identical layers will exhibit lower values of the DOP than the stacks with non-identical layers.