The phonon dispersions and scattering rates of the thermoelectric material Ag1−xSb1+xTe2+x (x = 0, 0.1, 0.2) were measured with inelastic neutron scattering, as function of both temperature T and off-stoichiometry x. In addition, detailed measurements of diffuse scattering were performed with both neutron and synchrotron x-ray diffraction. The results show that phonon scattering rates are large and weakly dependent on T or x, and the lattice thermal conductivity calculated from these scattering rates and group velocities is in good agreement with bulk transport measurements. We also find that the scattering rates and their temperature dependence cannot be accounted for with common models of phonon scattering by anharmonicity or point defects. The diffuse scattering measurements show a pervasive, complex signal, with several distinct components. In particular, broad superstructure reflections indicate a short-range ordering of the Ag and Sb cations on their sublattice. Single-crystal Bragg peak intensities also reveal large static atomic displacements, compatible with results from Rietveld refinement of neutron powder diffraction data. Our results indicate that a complex nanostructure, arising from multiple variants of nanoscale anisotropic superstructures of cations, and large atomic displacements, is likely responsible for the strong phonon scattering.