The unmanned aerial vehicle- (UAV-) assisted sub-6 GHz disaster relief networks cannot meet high-speed transmission requirements. In this paper, the millimeter wave (mmWave) frequency band is combined with the sub-6 GHz frequency band to build a high-speed UAV-assisted disaster relief network. However, the high propagation path loss of mmWave signals usually needs to be compensated by beamforming, where the ground-facing beam of each UAV is the desired receiving beam of ground user information. The different channels need to be allocated to a single UAV so that this kind of beam can be used simultaneously by different ground users to communicate with this UAV. Also, the other UAVs should reuse these channels as much as possible to save spectrum resources. In this paper, the beamforming training (BFT) mechanism is firstly used to obtain the signal-to-noise ratio (SNR) values of all possible links between ground terminals and UAVs, which are used to estimate these links’ energy efficiency. Then, an interference graph construction algorithm is proposed to identify the links that cannot be used simultaneously in the same channel according to the system energy efficiency. Finally, an iterative channel allocation algorithm is designed to allocate new channels to eliminate the edges of the interference graph, so that the links obtained by the BFT process can be used simultaneously as much as possible under the constraint of the number of channels. The simulation results show that our proposed scheme can achieve the shorter average convergence time, the higher data rate (or the lower data loss rate), and the higher energy efficiency.