Wireless communication capability is vital for remote transmission between unmanned aerial vehicle (UAV) coordination systems, but the communication link between highly dynamic UAV systems is unstable and severely disturbed due to its rapidly time-varying channel. Current channel estimation methods suffer from insufficient inter-symbol interference (ISI) and inter-carrier interference (ICI) suppression and inadequate noise filtering. Therefore, a collaborative channel estimation network (CoCENet) is proposed in this paper, and it can restrain the channel interference by capturing the amplitude–phase and time–frequency correlation at the same time. Moreover, CoCENet applies a multi-scale fusion strategy to optimize the purity of the estimated outcome. Our experiment results demonstrate that CoCENet has preferable performance in terms of the suppression of channel interference and noise in rapidly time-varying UAV systems in a complex environment without stationarity assumption. At a signal-to-noise ratio (SNR) of −10 dB, the mean square error (MSE) of CoCENet is improved by 1.7–2.3 dB compared to existing methods, and at a SNR of 20 dB, the MSE is improved by 1.1–2.2 dB.