Abstract. The narrow K-z relation of powerful radio galaxies in the Hubble K-band diagram is often attributed to the stellar populations of massive elliptical galaxies. Because it extends over a large range of redshifts (0 < z < 4), it is difficult to estimate masses at high redshifts by taking into account galaxy evolution. In the present paper, we propose to estimate the stellar masses of galaxies using the galaxy evolution model PÉGASE. We use star formation scenarios that successfully fit faint galaxy counts as well as z = 0 galaxy templates. These scenarios also predict spectra at higher z, used to estimate valid photometric redshifts. The baryonic mass of the initial gas cloud M bar is then derived. The K-z relation is remarkably reproduced by our evolutionary scenario for elliptical galaxies of baryonic mass M bar,max 1012 M , at all z up to z = 4. M bar,max is also the maximum mass limit of all types of galaxies. Using another initial mass function (IMF), even a top-heavy one, does not alter our conclusions. The high value of M bar,max observed at z > 4 implies that massive clouds were already formed at early epochs. We also find that the M bar,max limit is similar to the critical mass M crit of a self-gravitating cloud regulated by cooling (Rees & Ostriker 1977;Silk 1977). Moreover, the critical size r crit 75 kpc is remarkably close to the typical diameter of Lyα haloes surrounding distant radio galaxies. This confirms the validity of the method of baryonic mass determination based on the K-band luminosity. A puzzling question that remains to be answered is the short time-scale of mass-accumulation required to form such massive galaxies at z = 4. We discuss the dispersion of the K-z relation in terms of uncertainties on the mass limit. The link between the presence of the active nucleus and a large stellar mass is also discussed.