Recently, the application of polymer-based composites at cryogenic conditions has become a hot topic, especially in aerospace fields. At cryogenic temperature, the polymer becomes more brittle, and the adverse effect of thermal stress induced by temperature is more remarkable. In this paper, the research development of thermoset and thermoplastic polymers for cryogenic applications are all reviewed. This review considers the literature concerning: (a) the cryogenic performance of modified thermoset polymers and the improving mechanisms of the reported modification methods; (b) the cryogenic application potential of some commercial thermoplastic polymers and the cryogenic performance of modified thermoplastic polymers; (c) the recent advance in the use of polymer for special cryogenic environment-liquid oxygen. This paper provides a comprehensive overview of the research development of the polymer for cryogenic application. Moreover, future research directions have been proposed to facilitate its practical applications in aerospace.