This work investigates the dominant energy dissipations of the multi-frequency whispering gallery mode (WGM) resonators to provide an insight into the loss mechanisms of the devices. An extensive theory for each loss source was established and experimentally testified. The squeezed film damping (SFD) is a major loss for all the WGMs at atmosphere, which is distinguished from traditional bulk acoustic wave (BAW) resonators where the high-order modes suffer less from the air damping. In vacuum, the SFD is negligible, and the frequency-dependent Akhiezer damping (AKE) has significant effects on different order modes. For low-order WGMs, the AKE is limited, and the anchor loss behaves as the dominant loss. For high-order modes with an extended nodal region, the anchor loss is reduced, and the AKE determines the Q values. Substantial Q enhancements over four times and an excellent f × Q product up to 6.36 × 1013 at 7 K were achieved.