In this work, the experimental evidence of glass‐like phonon dynamics and thermal conductivity in a nanocomposite made of GeTe and amorphous carbon is reported, which is of interest for microelectronics, and specifically phase change memories. It is shown that, the total thermal conductivity is reduced by a factor of three at room temperature with respect to pure GeTe, due to the reduction of both electronic and phononic contributions. This latter, similarly to glasses, is small and weakly increasing with temperature between 100 and 300 K, indicating a mostly diffusive thermal transport and reaching a value of 0.86(7) Wm−1K−1 at room temperature. A thorough investigation of the nanocomposite's phonon dynamics reveals the appearance of an excess intensity in the low energy vibrational density of states, reminiscent of the Boson peak in glasses. These features can be understood in terms of an enhanced phonon scattering at the interfaces, due to the presence of elastic heterogeneities, at wavelengths in the 2–20 nm range. The findings confirm recent simulation results on crystalline/amorphous nanocomposites and open new perspectives in phonon and thermal engineering through the direct manipulation of elastic heterogeneities.