A general spin system with a nonaxially symmetric Hamiltonian containing Jx, Jz-linear and Jz-quadratic terms, widely used in many-body fermionic and bosonic systems and in molecular magnetism, is considered for the variations of general parameters describing intensity interaction changes of each of its terms. For this model Hamiltonian, a semiclassical energy surface (ES) is obtained by means of the coherent-state formalism. An analysis of this ES function, based on catastrophe theory, determines the separatrix in the control parameter space of the system Hamiltonian: the loci of singularities representing semiclassical phase transitions. Here we show that distinct regions of qualitatively different spectrum structures, as well as a singular behavior of quantum states, are ruled by this separatrix: here we show that the separatrix not only describes ground-state singularities, which have been associated with quantum phase transitions, but also reveals the structure of the excited spectrum, distinguishing different quantum phases within the parameter space. Finally, we consider magnetic susceptibility and heat capacity of the system at finite temperature, in order to study thermal properties and thermodynamical phase transitions in the perspective of the separatrix of this Hamiltonian system.