With the development of microelectronic devices having miniaturized and integrated electronic components, an efficient thermal management system with lightweight materials, which have outstanding thermal conductivity and processability, is becoming increasingly important. Recently, the use of polymer-based thermal management systems has attracted much interest due to the intrinsic excellent properties of the polymer, such as the high flexibility, low cost, electrical insulation, and excellent processability. However, most polymers possess low thermal conductivity, which limits the thermal management applications of them. To address the low thermal conduction of the polymer materials, many kinds of thermally conductive fillers have been studied, and the carbon-based polymer composite is regarded as one of the most promising materials for the thermal management of the electric and electronic devices. In addition, the next generation electronic devices require composite materials with various additional functions such as flexibility, low density, electrical insulation, and oriented heat conduction, as well as ultrahigh thermal conductivity. In this review, we introduce the latest papers on thermally conductive polymer composites based on carbon fillers with sophisticated structures to meet the above requirements. The topic of this review paper consists of the following four contents. First, we introduce the design of a continuous three-dimensional network structure of carbon fillers to reduce the thermal resistance between the filler–matrix interface and individual filler particles. Second, we discuss various methods of suppressing the electrical conductivity of carbon fillers in order to manufacture the polymer composites that meet both the electrical insulation and thermal conductivity. Third, we describe a strategy for the vertical alignment of carbon fillers to improve the through-plane thermal conductivity of the polymer composite. Finally, we briefly mention the durability of the thermal conductivity performance of the carbon-based composites. This review presents key technologies for a thermal management system of next-generation electronic devices.