We study the phonon-assisted intraband relaxation of electrons and holes confined in Si nanocrystals. The rates of relaxation processes are calculated as functions of the nanocrystal size and of the temperature. It occurs that the main contribution to the relaxation is provided by the mechanism where a single acoustic and a number of optical phonons, which is necessary to compensate the energy difference between the quantized charge carrier levels, are involved. We show that the phonon-assisted transitions between neighboring, size-quantized levels occur typically on a picosecond timescale, but vary over several orders of magnitude with the nanocrystal size. This results in a multi-exponential decay of the carrier populations averaged over an ensemble of the nanocrystals with a given size distribution. When the nanocrystal size is reduced and more than two phonons are required for the transition, there is a qualitative difference in the behavior of the transition probabilities between the electrons and the holes. Whereas the electron transition times strongly oscillate around approximately the same mean values in the picosecond range with some drops towards nanoseconds, there is a clearly pronounced tendency of the relaxation time increase into the nanosecond time range for the hole transitions when the nanocrystal size is decreased. The increase of the temperature leads to a moderate decrease of the relaxation times but does not change the picture qualitatively.