Experimental studies have shown that the boundary-layer flow over a rotating cone is susceptible to cross-flow and centrifugal instability modes of spiral nature, depending on the cone sharpness. For half-angles (ψ) ranging from propeller nose cones to rotating disks (ψ > 40• ), the instability triggers co-rotating vortices, whereas for sharp spinning missiles (ψ < 40• ), counter-rotating vortices are observed. In this paper we provide a mathematical description of the onset of co-rotating vortices for a family of cones rotating in quiescent fluid, with a view towards explaining the effect of ψ on the underlying transition of dominant instability. We investigate the stability of inviscid cross-flow modes (type I) as well as modes which arise from a viscous-Coriolis force balance (type II), using numerical and asymptotic methods. The influence of ψ on the number and orientation of the spiral vortices is examined, with comparisons drawn between our two distinct methods as well as with previous experimental studies. Our results indicate that increasing ψ has a stabilizing effect on both the type I and type II modes. Favourable agreement is obtained between the numerical and asymptotic methods presented here and existing experimental results for ψ > 40• . Below this half-angle we suggest that an alternative instability mechanism is at work, which is not amenable to investigation using the formulation presented here.