Thermoelectric (TE) materials are useful for the conversion of heat flux into electrical power and for heat removal and are, therefore, technologically important. Development of TE materials involves controlling their thermal and electrical transport, and both are sensitive to the finest features of the material microstructure. Major turning points recorded in TE material research are enabled due to adopting concepts from physical metallurgy. These classical approaches, originally developed to explain plastic deformation mechanisms in metallic systems, are now being employed to elucidate thermal and electrical transport in highly efficient TE materials, and serve as tools to manipulate their transport properties. The present Progress Report reviews recent studies combining experimental and computational approaches aiming to elucidate the role of 0D, 1D, 2D, and 3D lattice defects, and to harness them to enhance TE performance.