As a common but troublesome branch of stray light, ghost images in catadioptric optical systems greatly limit the performances of such optics, including panoramic annular lenses (PAL). A versatile and robust real-time characterization algorithm for ghost images in PAL systems is proposed, with a new ray behavior quantifying index. It characterizes any kind of ghost image in the catadioptric unit with a high reliability, regardless of the actual existence of the ghost images. With newly extracted common feature and properly designed numerical methods specified to PAL systems locating representative rays, the evaluation runs almost instantly, allowing itself to be implemented in optical design process to perform real-time analyses on and to suppress ghost images. Multiple examples show the ability of the method on analyzing ghost image behaviors and achieving ghost-image-suppressed PAL designs. Along with further discussions, the potential of extending the method and its possible wider prospects can be reflected.