We find a normal form which describes the high energy dynamics of a class of piecewise smooth Fermi-Ulam ping pong models. Depending on the value of a single real parameter, the dynamics can be either hyperbolic or elliptic. In the first case, we prove that the set of orbits undergoing Fermi acceleration has zero measure but full Hausdorff dimension. We also show that for almost every orbit, the energy eventually falls below a fixed threshold. In the second case, we prove that, generically, we have stable periodic orbits for arbitrarily high energies and that the set of Fermi accelerating orbits may have infinite measure.