In this work, the effect of Si doping on InAs/GaAs quantum dot solar cells with AlAs cap layers is studied. The AlAs cap layers suppress the formation of the wetting layer during quantum dot growth. This helps achieve quantum dot state filling, which is one of the requirements for strong sub-bandgap photon absorption in the quantum dot intermediate band solar cell, at lower Si doping density. Furthermore, the passivation of defect states in the quantum dots with moderate Si doping is demonstrated, which leads to an enhancement of the carrier lifetime in the quantum dots, and hence the open-circuit voltage.