Phase and amplitude modes, also called polariton modes, are emergent phenomena that manifest across diverse physical systems, from condensed matter and particle physics to quantum optics. We study their behavior in an anisotropic Dicke model that includes collective matter interactions. We study the low-lying spectrum in the thermodynamic limit via the Holstein–Primakoff transformation and contrast the results with the semi-classical energy surface obtained via coherent states. We also explore the geometric phase for both boson and spin contours in the parameter space as a function of the phases in the system. We unveil novel phenomena due to the unique critical features provided by the interplay between the anisotropy and matter interactions. We expect our results to serve the observation of phase and amplitude modes in current quantum information platforms.