The development of effective thermally conductive rubber nanocomposites for heat management represents a tricky point for several modern technologies, ranging from electronic devices to the tire industry. Since rubber materials generally exhibit poor thermal transfer, the addition of high loadings of different carbon-based or inorganic thermally conductive fillers is mandatory to achieve satisfactory heat dissipation performance. However, this dramatically alters the mechanical behavior of the final materials, representing a real limitation to their application. Moreover, upon fillers’ incorporation into the polymer matrix, interfacial thermal resistance arises due to differences between the phonon spectra and scattering at the hybrid interface between the phases. Thus, a suitable filler functionalization is required to avoid discontinuities in the thermal transfer. In this challenging scenario, the present review aims at summarizing the most recent efforts to improve the thermal conductivity of rubber nanocomposites by exploiting, in particular, inorganic and hybrid filler systems, focusing on those that may guarantee a viable transfer of lab-scale formulations to technological applicable solutions. The intrinsic relationship among the filler’s loading, structure, morphology, and interfacial features and the heat transfer in the rubber matrix will be explored in depth, with the ambition of providing some methodological tools for a more profitable design of thermally conductive rubber nanocomposites, especially those for the formulation of tires.