With the development of electronic equipment components in the direction of integration, miniaturization, and intelligence, their increasingly high heat dissipation requirements pose a high challenge to the thermal conductivity (TC) of thermal interface materials (TIMs). Polymer-based composite materials with high TC have gradually been favored because of their good processing performance, low cost, and low density. It is important to summarize the relationship between the problems of low heat conduction and their solutions in relation to polymer-based composite materials. For this purpose, this review comprehensively discusses the basic mechanisms of heat transfer inside polymer-based TIMs, the current challenges, and future prospects for improving TC. Strategies involving surface modification and network construction can reduce interfacial thermal resistance and enhance heat conduction.