We show that hot phonons emitted in energy conversion or resistive processes can be converted to electric potential in heterobarrier structures. Using phonon and electron interaction kinetics and self-consistent ensemble Monte Carlo, we find the favorable conditions for unassisted absorption of hot phonons and design graded heterobarriers for their direct conversion into electric energy. Tandem barriers with nearly optical-phonon height allow for substantial potential gain without current loss. We find that 19% of hot phonons can be harvested with an optimized GaAs/Al x Ga 1−x As barrier structure over a range of current and electron densities, thus enhancing the overall energy conversion efficiency and reducing waste heat.