Polarization holography has the unique capacity to record and retrieve the amplitude, phase, and polarization of light simultaneously in a polarization-sensitive recording material and has attracted widespread attention. Polarization holography is a noteworthy technology with potential applications in the fields of high-capacity data storage, polarization-controlled optical elements, and other related fields. The choice of its high-performance materials is particularly important. To further develop polarization holography applications and improve the quality of the information recorded (i.e., material sensitivity and resolution), a deeper understanding of such materials is needed. We present an overview of the polarization-sensitive materials, which introduced polarization holographic technology and the development of polarization holographic materials. The three main types of polarization holographic materials are described, including azopolymer materials, photopolymer material, and photorefractive polymer material. We examine the key contributions of each work and present many of the suggestions that have been made to improve the different polarization-sensitive photopolymer materials.