We report on a systematic study of nanomechanical dissipation in high-frequency (≈300  MHz) gallium arsenide optomechanical disk resonators, in conditions where clamping and fluidic losses are negligible. Phonon-phonon interactions are shown to contribute with a loss background fading away at cryogenic temperatures (3 K). Atomic layer deposition of alumina at the surface modifies the quality factor of resonators, pointing towards the importance of surface dissipation. The temperature evolution is accurately fitted by two-level systems models, showing that nanomechanical dissipation in gallium arsenide resonators directly connects to their microscopic properties. Two-level systems, notably at surfaces, appear to rule the damping and fluctuations of such high-quality crystalline nanomechanical devices, at all temperatures from 3 to 300 K.