Herein, it is demonstrated that optical excitation of InAs quantum dots (QDs) embedded directly in an InP matrix can be mediated via states in a quaternary compound constituting an InP/InGaAlAs bottom distributed Bragg reflector (DBR) and native defects in the InP matrix. It does not only change the carrier relaxation in the structure but could also lead to the imbalanced occupation of QDs with charge carriers, because the band structure favors the transfer of holes. Thermal activation of carrier transfer can be observed as an increase in the emission intensity versus temperature for excitation powers below saturation on the level of both an inhomogeneously broadened QD ensemble and single QD transitions. That increase in the QD emission is accompanied by a decrease in the emission from the InGaAlAs layer at low temperatures. Finally, carrier transfer between the InGaAlAs layer of the DBR and the InAs/InP QDs is directly proven by the photoluminescence excitation spectrum of the QD ensemble. The reported carrier transfer can increase the relaxation time of carriers into the QDs and thus be detrimental to the coherence properties of single and entangled photons. It is important to take it into account while designing QD‐based devices.