We examine the temperature dependence of the visibility of the excitonic Aharonov-Bohm peak in type-II quantum dots. We obtain a functional temperature dependence, which is similar to that determined by transport experiments, namely, with T -1 term due to electron-electron collisions and with T -3 term due to electron-phonon interactions. However, the magnitude of the latter term is much smaller than that for the transport electrons and similar to the interaction strength of the exciton-phonon coupling. Such suppressed electron-phonon interaction ushers a way for all-optical studies of decoherence processes in semiconductor nanostructures as other dephasing mechanisms become more pronounced.