Er-doped InAs quantum dots (QDs) embedded in strain-relaxed InGaAs barriers, which exhibit an extremely short carrier decay time of 3 ps due to the nonradiative process, are superior materials for ultrafast all-optical switches using a GaAs/AlAs multilayer cavity. The intensity of the nonlinear signal due to the absorption saturation in the 20-layer stack of the Er-doped QDs was increased by increasing the In composition in the strain-relaxed InGaAs barriers while keeping the extremely short decay time. The QD cavity structure, which consisted of GaAs/AlAs distributed Bragg reflector (DBR) multilayers and a half-wavelength cavity layer containing two layers of the Er-doped QDs was grown by molecular beam epitaxy. The transmission change signal was clearly observed in the time-resolved measurements at the cavity mode wavelength of 1.55 µm. The response time of 4 ps was observed for the Er-doped QD cavity, which was much shorter than that (12 ps) for the undoped QD cavity.