The Poincaré sphere is a graphical representation in a three-dimensional space for the polarization of light. Similarly, an optical element with spatially varying birefringence can be represented by a surface on a four-dimensional "Poincaré hypersphere." A projection of this surface onto the traditional Poincaré sphere provides an intuitive geometric description of the polarization transformation performed by the element, as well as the induced geometric phase. We apply this formalism to quantify the effects of birefringence on the image quality of an optical system.