In this paper, we propose a new physico-mathematical model for the Rayleigh–Benard convection in a gas-vapor mixture of oxygen and cyclohexane, for which the temperature dependence of density has a maximum. Linear stability analysis was performed; two limiting cases where the inversion parameter tends to its limiting values are studied analytically. Formulas are obtained for the growth increment and decay, the neutral curve, and the boundary of the instability region in the wave plane.