The combined effect of rotation and magnetic field on the onset of convection in a horizontal layer of electrically conducting nanofluid is studied. Three cases of free–free, rigid–free, and rigid–rigid boundaries are considered. The two important effects of Brownian motion and thermophoresis are included in the model of nanofluid. The linear stability theory is employed, and the resulting eigenvalue problem is solved analytically for free–free boundaries and numerically for rigid–free and rigid–rigid boundaries using the Galerkin technique. Numerical results are presented for alumina–water nanofluid. The effects of magnetic field, rotation, and various nanofluid parameters such as nanoparticle Rayleigh number, modified diffusive ratio, and Lewis number on the onset of convection are analyzed, and results are depicted graphically. It is found that the critical Rayleigh number is lower for nanofluid in comparison with the regular fluid at the same values of Chandrasekhar number and Taylor number. The sufficient condition for the non‐existence of overstability is also obtained. © 2014 Curtin University of Technology and John Wiley & Sons, Ltd.