As the core unit of energy storage equipment, high voltage pulse capacitor plays an indispensable role in the field of electric power system and electromagnetic energy related equipment. The mostly utilized polymer materials are metallized polymer thin films, which are represented by biaxially oriented polypropylene (BOPP) films, possessing the advantages including low cost, high breakdown strength, excellent processing ability, and self-healing performance. However, the low dielectric constant (εr < 3) of traditional BOPP films makes it impossible to meet the demand for increased high energy density. Controlled/living radical polymerization (CRP) and related techniques have become a powerful approach to tailor the chemical and physical properties of materials and have given rise to great advances in tuning the properties of polymer dielectrics. Although organic-inorganic composite dielectrics have received much attention in previous studies, all-organic polymer dielectrics have been proven to be the most promising choice because of its light weight and easy large-scale continuous processing. In this short review, we begin with some basic theory of polymer dielectrics and some theoretical considerations for the rational design of dielectric polymers with high performance. In the guidance of these theoretical considerations, we review recent progress toward all-organic polymer dielectrics based on two major approaches, one is to control the polymer chain structure, containing microscopic main-chain and side-chain structures, by the method of CRP and the other is macroscopic structure design of all-organic polymer dielectric films. And various chemistry and compositions are discussed within each approach.