Context. The form and evolution of the galaxy stellar mass function (GSMF) at high redshifts provide crucial information on star formation history and mass assembly in the young Universe, close or even prior to the epoch of reionization. Aims. We used the unique combination of deep optical/near-infrared/mid-infrared imaging provided by HST, Spitzer, and the VLT in the CANDELS-UDS, GOODS-South, and HUDF fields to determine the GSMF over the redshift range 3.5 ≤ z ≤ 7.5. Methods. We used the HST WFC3/IR near-infrared imaging from CANDELS and HUDF09, reaching H 27−28.5 over a total area of 369 arcmin 2 , in combination with associated deep HST ACS optical data, deep Spitzer IRAC imaging from the SEDS programme, and deep Y and K-band VLT Hawk-I images from the HUGS programme, to select a galaxy sample with high-quality photometric redshifts. These have been calibrated with more than 150 spectroscopic redshifts in the range 3.5 ≤ z ≤ 7.5, resulting in an overall precision of σ z /(1 + z) ∼ 0.037. With this database we have determined the low-mass end of the high-redshift GSMF with unprecedented precision, reaching down to masses as low as M * ∼ 10 9 M at z = 4 and ∼6 × 10 9 M at z = 7. Results. We find that the GSMF at 3.5 ≤ z ≤ 7.5 depends only slightly on the recipes adopted to measure the stellar masses, namely the photometric redshifts, the star formation histories, the nebular contribution, or the presence of AGN in the parent sample. The low-mass end of the GSMF is steeper than has been found at lower redshifts, but appears to be unchanged over the redshift range probed here. Meanwhile the high-mass end of the GSMF appears to evolve primarily in density, although there is also some evidence of evolution in characteristic mass. Our results are very different from previous mass function estimates based on converting UV galaxy luminosity functions into mass functions via tight mass-to-light relations. Integrating our evolving GSMF over mass, we find that the growth of stellar mass density is barely consistent with the time-integral of the star formation rate density over cosmic time at z > 4. Conclusions. These results confirm the unique synergy of the CANDELS+HUDF, HUGS, and SEDS surveys for the discovery and study of moderate/low-mass galaxies at high redshifts, and reaffirm the importance of space-based infrared selection for the unbiased measurement of the evolving GSMF in the young Universe.