We explore the ferromagnetic quantum critical point in a three-dimensional semimetallic system with upward-and downward-dispersing bands touching at the Fermi level. Evaluating the static spin susceptibility to leading order in the coupling between the fermions and the fluctuating ferromagnetic order parameter, we find that the ferromagnetic quantum critical point is masked by an incommensurate, longitudinal spin density wave phase. We first analyze an idealized model which, despite having strong spin-orbit coupling, still possesses O(3) rotational symmetry generated by the total angular momentum operator. In this case, the direction of the incommensurate spin density wave propagation can point anywhere, while the magnetic moment is aligned along the direction of propagation. Including symmetry-allowed anisotropies in the fermion dispersion and the coupling to the order parameter field, however, the ordering wavevector instead breaks a discrete symmetry and aligns along either the [111] or [100] direction, depending on the signs and magnitudes of these two types of anisotropy.
PACS numbers:For decades it has been understood that electron interactions may lead to interesting consequences at low energy scales in three-dimensional (3D) semimetallic systems, in which spin-orbit coupling combines with cubic crystalline symmetries, leading to a band degeneracy at the Fermi energy. 1-8 Such a band structure has long been known to occur in HgTe and α-Sn (gray tin), and has possibly been discovered more recently in the pyrochlore Pr 2 Ir 2 O 7 . 9-14 Since the low-energy effective theory for these systems was first developed by Luttinger 1 , it has been argued using various approaches that interactions could lead to an excitonic instability at low temperatures 2,6 , or that the system may be described by an exotic non-Fermi liquid phase, characterized by nontrivial power-law scaling of various physical quantities with temperature. [3][4][5] In more recent work, the properties of such a system near an Ising antiferromagnetic quantum critical point were explored, and the critical theory was found to be governed by unusual critical exponents and emergent spatial anisotropy of the fermion dispersion. 7 The theory describing the quantum phase transition into an insulating nematic phase has also been developed recently. 8In this work we explore the fate of a 3D semimetal in the vicinity of a ferromagnetic (FM) quantum critical point. Working at zero temperature and approaching the quantum critical point from the paramagnetic side, we find that it is unstable toward an incommensurate spin density wave (SDW) phase. In the most symmetric version of the theory, there is a combined O(3) rotational symmetry for spin and spatial degrees of freedom, which is spontaneously broken by the SDW wavevector. The O(3) symmetry is reduced to a discrete symmetry by either of two sources of anisotropy: the Yukawa coupling of the fermions to the fluctuating FM field, or the dispersion of the fermions themselves.(Throughout this work, by anisotro...