The materials with low thermal conductivity (<em>κ</em>) are both fundamentally interesting and technologically important in applications related to thermal energy conversion and thermal management, such as thermoelectric conversion devices, thermal barrier coatings, and thermal storage. Therefore, understanding the physical mechanisms of glass-like heat conduction in crystalline materials is essential for the development and design of low-<em>κ</em> materials. In this paper, the microscopic phonon mechanisms of glass-like low <em>κ</em> in binary simple crystal Yb<sub>3</sub>TaO<sub>7</sub> with fluorite structure are revealed by using the equilibrium molecular dynamics, phonon spectral energy density, and lattice dynamics. Meanwhile, the weberite-structured Yb<sub>3</sub>TaO<sub>7</sub> is also included for comparison. The calculated <em>κ</em> indicates that fluorite Yb<sub>3</sub>TaO<sub>7</sub> has a glass-like low <em>κ</em> while weberite Yb<sub>3</sub>TaO<sub>7</sub> has a crystal <em>κ</em>. Such a low <em>κ</em> in fluorite Yb<sub>3</sub>TaO<sub>7</sub> is mainly due to the large difference in interatomic force between O-Yb and O-Ta. This differing atomic bonding can significantly soften the phonon modes and thus limit phonon transport. To further describe the microscopic phonon thermal conduction, the single-channel model based on the phonon gas model is first applied to calculate the total <em>κ</em>. However, the single-channel model significantly underestimates the <em>κ</em>, suggesting the presence of non-normal phonons in Yb<sub>3</sub>TaO<sub>7</sub>. Based on this, vibrational mode decomposition is conducted throughout the entire phonon spectrum of fluorite- and weberite-type Yb<sub>3</sub>TaO<sub>7</sub>. It is found that most modes in fluorite Yb<sub>3</sub>TaO<sub>7</sub> fall in the Ioffe—Regel regime and exhibit a strongly diffusive nature. Such diffusive modes cannot be described by the phonon gas model. Based on the decomposed phonon modes, the dual-channel model involving diffusive and propagating modes is applied to describe the phonon thermal conduction, by which the obtained results agree well with the experimental values. The vast majority (>90%) of heat in fluorite Yb<sub>3</sub>TaO<sub>7</sub> is found to be transported by diffusive modes rather than propagating modes. Consequently, the <em>κ</em> of fluorite Yb<sub>3</sub>TaO<sub>7</sub> increases with the increase in temperature, exhibiting a unique glass-like nature. In particular, contrary to conventional wisdom, the optical phonon modes in fluorite Yb<sub>3</sub>TaO<sub>7</sub> play a significant or even decisive role in thermal conduction, which could add a new physical dimension to the regulation of <em>κ</em> in solid materials. Overall, the new understanding of the link between chemical bonding and glass-like <em>κ</em> could contribute to the development and design of low-<em>κ</em> materials.