Recently, the possibility of implementing a cavity-enhanced off-resonant Raman quantum memory in an ensemble of silicon-vacancy centers in diamond has been studied (Kalachev et al 2019 Laser Phys. 29 104001). It was shown that the signal-to-noise ratio can significantly exceed unity for short single-photon input pulses if the orbital level splitting in the ground state of the color centers is significantly enhanced by strain. The latter results in decreasing the coherence time between the two available orbital branches of the ground state, though. In the present work, we consider the possibility of increasing the storage time due to the use of nanodiamonds. It is shown that suppression of direct electron-phonon transitions in the diamond nanocrystals makes it possible to increase the coherence time of both orbital and spin qubits by orders of magnitude.