A variety of crystals contain quasi-one-dimensional substructures, which yield distinctive electronic, spintronic, optical and thermoelectric properties. There is a lack of understanding of the lattice dynamics that influences the properties of such complex crystals. Here we employ inelastic neutron scatting measurements and density functional theory calculations to show that numerous low-energy optical vibrational modes exist in higher manganese silicides, an example of such crystals. These optical modes, including unusually low-frequency twisting motions of the Si ladders inside the Mn chimneys, provide a large phase space for scattering acoustic phonons. A hybrid phonon and diffuson model is proposed to explain the low and anisotropic thermal conductivity of higher manganese silicides and to evaluate nanostructuring as an approach to further suppress the thermal conductivity and enhance the thermoelectric energy conversion efficiency. This discovery offers new insights into the structure-property relationships of a broad class of materials with quasi-one-dimensional substructures for various applications.