Dielectric capacitors are widely used in pulse power systems, electric vehicles, aerospace, and defense technology as they are crucial for electronic components. Compact, lightweight, and diversified designs of electronic components are prerequisites for dielectric capacitors. Additionally, wide temperature stability and high energy storage density are equally important for dielectric materials. Ferroelectric materials, as special (spontaneously polarized) dielectric materials, show great potential in the field of pulse power capacitors having high dielectric breakdown strength, high polarization, low-temperature dependence and high energy storage density. The first part of this review briefly introduces dielectric materials and their energy storage performance. The second part elaborates performance characteristics of various ferroelectric materials in energy storage and refrigeration based on electrocaloric effect and briefly shed light on advantages and disadvantages of various common ferroelectric materials. Especially, we summarize the polarization effects of underlying substrates (such as GaN and Si) on the performance characteristics of ferroelectric materials. Finally, the review will be concluded with an outlook, discussing current challenges in the field of dielectric materials and prospective opportunities to assess their future progress.