We introduce a group-theoretical extension of the Dicke model, which describes an ensemble of two-level atoms interacting with a finite radiation field. The latter is described by a spin model whose main feature is that it possesses a maximum number of excitations. The approach adopted here leads to a nonlinear extension of the Dicke model that takes into account both the intensity dependent coupling between the atoms and the radiation field and an additional nonlinear Kerr-like or Pösch–Teller-like oscillator term, depending on the degree of nonlinearity. We use the energy surface minimization method to demonstrate that the extended Dicke model exhibits a quantum phase transition, and we analyze its dependence upon the maximum number of excitations of the model. Our analysis is carried out via three methods: through mean-field analysis (i.e., by using the tensor product of coherent states), by using parity-preserving symmetry-adapted states (using the critical values obtained in the mean-field analysis and numerically minimizing the energy surface), and by means of the exact quantum solution (i.e., by numerically diagonalizing the Hamiltonian). Possible connections with the qp-deformed algebras are also discussed.