Polarized catadioptric systems (also called folded optics, pancake lenses) are a viable solution for virtual reality (VR) head-mount display applications. Reasons include its high image resolution, wide field-of-view, smaller physical size and lighter weight compared to the conventional refractive optics counterparts. One of the major challenges with folded optics systems is to reduce the light intensities of the ghost images and spurious halos due to multi-bounces, hence improve the overall image contrast. In this paper we discuss a high-quality polarized catadioptric system and present a ghost analysis method and the corresponding results. It can be shown that the many ghosts in the image plane can be coarsely categorized into "local ghosts" and "distant ghosts" according to their proximity to the image. The local ghosts or halos are close to the image itself and they directly decrease image contrast; whereas the distant ghosts are located diagonally away from the image and they can create some distracting faint images under certain display content. The origin of each individual ghost and halo can be traced back to its contributing optical components, which provides targeted information to directly mitigate them on a component level. Experimental results are shown to demonstrate the validity and efficacy of this approach. Furthermore, the ghost analysis information can also be used as additional constraints in the lens design software to perform new rounds of optimization on a system level.