In this paper, we use the Buongiorno's mathematical model for studying numerically the convective instability which is induced by a purely internal heating in a rotating medium confined between two isothermal surfaces and filled of a Newtonian nanofluid layer (water + alumina) in the case where the radiation heat transfer mode is taken into consideration, such that the nanoparticle flux is zero on the boundaries. The linear study which was achieved in this investigation shows that the thermal stability of nanofluids depends of the Coriolis forces generated by the rotation of the system, the thermal radiation parameter, the Brownian motion, the thermophoresis of nanoparticles and other thermophysical properties of nanoparticles. The studied problem will be solved by converting our boundary value problem to an initial value problem, after this step we will approach numerically the searched solutions using the power series method (PSM).