The optical performance of a panoramic annular lens (PAL) might be severely deteriorated by stray light due to the catadioptric optical structure and the ultra-large field of view (FoV). A comprehensive research on stray light in PAL systems with experimental validation is carried out. The fundamental cause, forming light path, impact on the image plane, and first-hand solutions of all visible types of stray light are obtained and discussed. By analyzing some specific light paths, we also introduce the prerequisite of a real-time evaluation and suppressing algorithm for stray light in a PAL system that purely operates in optical design software.