Stability of a spin-triplet nematic state near to a quantum critical point

We analyze a model of itinerant electrons interacting through a quadrupole density-density repulsion in three dimensions. At the mean field level, the interaction drives a continuous Pomeranchuk instability towards d-wave, spin-triplet nematic order, which simultaneously breaks the SU(2) spin-rotation and spatial rotational symmetries. This order results in spin antisymmetric, elliptical deformations of the Fermi surfaces of up and down spins. We show that the effects of quantum fluctuations are similar to those in metallic ferromagnets, rendering the nematic transition first-order at low temperatures. Using the fermionic quantum order-by-disorder approach to self-consistently calculate fluctuations around possible modulated states, we show that the first-order transition is pre-empted by the formation of a nematic state that is intertwined with a helical modulation in spin space. Such a state is closely related to d-wave bond density wave order in square-lattice systems. Moreover, we show that it may coexist with a modulated, p-wave superconducting state.