A new approach for the calculation of angular momentum projected potential energy surfaces (AMPPESs) is proposed which combines the projected shell model with a quadrupole constrained relativistic Hartree-Bogoliubov (RHB) theory in which the NL3 effective interaction is chosen for the relativistic mean-field effective Lagrangian and a separable Gogny D1S interaction for the pairing force (QCRHB-NL3 + separable Gogny D1S force theory). We apply this approach to compute the AMPPESs of 80,82,84 Zr nuclei up to high spins and investigate the spin-induced shape transitions and decay out of the superdeformed (SD) bands in these nuclei. We find that the shape transitions occur in 80 Zr and 84 Zr, which are driven by the rotational alignments of the nucleons in the 1g 9/2 orbitals, and a strong shape mixing happens in 82 Zr. Moreover, it is shown that the barrier separating the SD states and normal deformed (or spherical) states becomes lower and narrower for 82 Zr and 84 Zr at high spins, indicating that the decay out of the SD bands could occur at high spins. For 80 Zr, however, there is no decay out of the SD band because the barrier is so high and thick. Meanwhile, the QCRHB-NL3 + separable Gogny D1S force theory is employed to calculate the ground-state potential energy surfaces and the single-particle levels of these nuclei, which in turn are used to determine and analyze the equilibrium shapes and discuss the shape coexistence of these nuclei. In addition, this theory is compared with other state-of-the-art mean-field theories to justify its use to study the ground-state potential energy surfaces of 80,82,84 Zr.