A recent experimental study (Phys. Rev. Appl. 9, 024023, 2018) on paramagnetic CeCo3 finds that Magnesium alloying induces a ferromagnetic transition with intrinsic properties large enough for permanent magnet applications. Here we explain these surprising results via a first principles study of the electronic structure and magnetism of Magnesium-alloyed CeCo3. We find the origin of this Magnesium-induced ferromagnetic transition to be Stoner physics -the substantial increase in the Fermi-level density-of-states N (EF ) with Mg alloying. Our calculations suggest that both Ce and Co atoms are important for generating large magnetic anisotropy suggesting the viability of Co-3d, and Ce-4f interaction for the generation of magnetic anisotropy in magnetic materials. These results offer a new route to the discovery of ferromagnetic materials and provide fundamental insight into the magnetic properties of these alloys.