Pancake optics (also known as polarization-based catadioptric system) has been widely used as the imaging lens for virtual reality (VR) and mixed reality (MR) headsets because of its compact formfactor and excellent image quality. However, such a folded pancake optics not only dramatically lowers the optical efficiency to 25% because of the utilization of a half mirror, but also suffers from ghost images due to the stray light from multiple surface reflections and imperfect polarization control inside the optical system. In this paper, the origins including the light paths of the ghost images are explored by experiment and then analyzed by simulation. The effect of different incident angles on the intensity of each ghost is also investigated.