We report on the shape and polarization control of site-controlled multiple and single InAs quantum dots (QDs) on InP pyramids grown by selective-area metal-organic vapor phase epitaxy. With increasing growth temperature the QDs elongate causing strong linear polarization of the photoluminescence. With reduced pyramid base/pyramid top area/QD number, the degree of polarization decreases, attributed to the symmetric pyramid top, reaching zero for single QDs grown at lower temperature. This control of linear polarization is important for entangled photon sources operating in the 1.55 μm wavelength region.