Nonequilibrium steady states of vibrated inelastic frictionless spheres are investigated in quasi-twodimensional confinement via molecular dynamics simulations. The phase diagram in the densityamplitude plane exhibits a fluidlike disordered and an ordered phase with threefold symmetry, as well as phase coexistence between the two. A dynamical mechanism exists that brings about metastable traveling clusters and at the same time stable clusters with anisotropic shapes at low vibration amplitude. Moreover, there is a square bilayer state which is connected to the fluid by BKTHNY-type two-step melting with an intermediate tetratic phase. The critical behavior of the two continuous transitions is studied in detail. For the fluid-tetratic transition, critical exponents ofγ = 1.73, η4 ≈ 1/4, and z = 2.05 are obtained.