This study focused on uncovering the relationship among nanofiller, crystallization behavior, and dielectric property of polymer composites. The effects of carbon nanofibers (CNFs) and heat treatment on the crystalline structures and dielectric properties of the semi‐crystalline polymers were analyzed by using high density polyethylene (HDPE) as a matrix, which is a representative of non‐polar polymer and contains only one crystal structure. The experimental results showed that the degree of crystallinity, size distribution of crystallity, and relative amount of different crystal planes in the HDPE matrix were changing due to the addition of CNFs. With the increase of CNF loading, the dielectric constant, dielectric loss and AC conductivity of the HDPE composites were increased, presenting a typical percolation characteristic, and the dependence of the dielectric constant on frequency became more obvious. All kinds of electronic transmission, polarization effect, and relaxation behaviors in CNF/HDPE composite system were deeply analyzed. After heat treatment, the degree of crystallinity of HDPE composites was decreased with the enhanced cooling rate. For the CNF/HDPE composites with nanofiller content slightly higher than the percolation threshold, the significant increase of the dielectric constant and the dramatical reduction of the dielectric loss over a wide frequency range were realized simultaneously through rapid cooling treatment. The research indicated that a general commercial polymer material with excellent dielectric properties, which exhibited a high dielectric constant and a low dielectric loss, can be obtained by a simple technical approach different from traditional fabrication method of threshold composites.