X-band electron spin resonance (ESR) spectroscopy has been applied to the study of molecular rotation of O2 molecules in isotopic solid hydrogen, D2 and HD. ESR signal of the O2 molecules in hindered rotational states has been observed, and its pressure dependence has been measured up to 19 MPa. Although molar volume of solid hydrogen decreases, the O2 rotation has become less hindered with the increase in pressure and isotope substitution from HD to D2. These effects of pressure and isotope substitution suggest that the potential barrier for the O2 rotation is mainly produced by distortion of O2 cages in D2 and HD. Since solid hydrogen becomes less compressible with the increase in pressure and the isotope substitution from HD to D2, the O2 rotation becomes less hindered in the less distorted O2 cages.