Abstract. In this paper, the processes of electromagnetic radiation generation as a result of the interaction of a relativistic electron beam with hydrogen and argon plasma are studied on the basis of numerical modeling by the particle-in-cells method (PIC). Series of numerical experiments for different background plasma parameters, beam and magnetic field have been performed using modern computer systems with massively parallel architecture. Estimates of the radiation efficiency for both the initially homogeneous plasma and for longitudinal density modulation are obtained. It is shown that the change in the plasma density due to the development of the modulation instability makes it possible to increase substantially the power of the generated sub-THz radiation. The parameters used in numerical experiments correspond to the conditions of laboratory experiments on GOL-3 facility (BINP SB RAS, Novosibirsk, Russia).