We study the performances of gaseous and two-phase (liquid-gas) cryogenic detectors of ionizing radiation based on gas electron multipliers (GEMs) and operated in an avalanche mode in pure noble gases. The gas amplification in He, Ar and Kr is systematically studied at low temperatures, using triple-GEM multipliers. High gains, exceeding 10 4 , were obtained in these gases in the range of 120-300 K. Stable electron avalanching was demonstrated in a saturated Kr vapor in the two-phase mode. These results are relevant for understanding basic mechanisms of electron avalanching at low temperatures and for applications in cryogenic particle detectors, in particular in dark matter and solar neutrino detectors.