The figure of merit of thermoelectric materials can be increased by suppressing the lattice thermal conductivity without degrading electrical properties. Phonons are the carriers for lattice thermal conduction, and their transport can be impeded by nanostructuring, owing to the recent progress in nanotechnology. The key question for further improvement of thermoelectric materials is how to realize ultimate structure with minimum lattice thermal conductivity. From spectral viewpoint, this means to impede transport of phonons in the entire spectral domain with noticeable contribution to lattice thermal conductivity that ranges in general from subterahertz to tens of terahertz in frequency. To this end, it is essential to know how the phonon transport varies with the length scale, morphology, and composition of nanostructures, and how effects of different nanostructures can be mutually adopted in view of the spectral domain. Here we review recent advances in analyzing such spectral impedance of phonon transport on the basis of various effects including alloy scattering, boundary scattering, and particle resonance.