A theoretical model of the electromagnetic H-wave interaction with a semiconductor nanolayer, the thickness of which can be comparable to or less than charge carrier de Broglie wavelength is constructed. We assume the frequency range of the electromagnetic wave to be much less than the plasma frequency. Analytical expressions are derived for optical coefficients as the functions of the dimensionless thickness, electromagnetic wave frequency and incidence angle, chemical potential, and surface roughness parameters. The results derived for the limiting cases of degenerate and nondegenerate electron gas are analyzed. Keywords: nanolayer, Liouville equation, de Broglie wavelength, Soffer model, optical coefficients.