We study the dynamics of photoexcited electrons and holes in single negatively charged CdSe/ZnSe quantum dots with two-color femtosecond pump-probe spectroscopy. An initial characterization of the energy level structure is performed at low temperatures and magnetic fields of up to 5 T. Emission and absorption resonances are assigned to specific transitions between few-fermion states by a theoretical model based on a configuration interaction approach. To analyze the dynamics of individual charge carriers, we initialize the quantum system into excited trion states with defined energy and spin. Subsequently, the time-dependent occupation of the trion ground state is monitored by spectrally resolved differential transmission measurements. We observe subpicosecond dynamics for a hole excited to the D shell. The energy dependence of this D-to-S shell intraband transition is investigated in quantum dots of varying size. Excitation of an electron-hole pair in the respective p shells leads to the formation of singlet and triplet spin configurations. Relaxation of the p-shell singlet is observed to occur on a time scale of a few picoseconds. Pumping of p-shell triplet transitions opens up two pathways with distinctly different scattering times. These processes are shown to be governed by the mixing of singlet and triplet states due to exchange interactions enabling simultaneous electron and hole spin flips. To isolate the relaxation channels, we align the spin of the residual electron by a magnetic field and employ laser pulses of defined helicity. This step provides ultrafast preparation of a fully inverted trion ground state of the quantum dot with near unity probability, enabling deterministic addition of a single photon to the probe pulse. Therefore our experiments represent a significant step towards using single quantum emitters with well-controled inversion to manipulate the photon statistics of ultrafast light pulses.