In the field of femtomagnetism magnetic matter is controlled by ultrafast laser pulses; here we show that coupling phonon excitations of the nuclei to spin and charge leads to femto-phono-magnetism, a powerful route to control magnetic order at ultrafast times. With state-of-the-art theoretical simulations of coupled spin-charge-and lattice-dynamics we identify strong non-adiabatic spinphonon coupled modes that dominate early time spin dynamics. Activating these phonon modes we show leads to an additional (up to 25%) loss of moment in FePt occurring within 40 femtoseconds of the pump laser pulse. Underpinning this enhanced ultrafast loss of spin moment we identify a physical mechanism in which minority spin-current drives an enhanced inter-site minority charge transfer, in turn promoting increased on-site spin flips. Our finding demonstrates that the nuclear system, often assumed to play only the role of an energy sink aiding long time re-magnetisation of the spin system, can play a profound role in controlling femtosecond spin-dynamics in materials.