Millimeter wave (mmWave)-based unmanned aerial vehicle (UAV) communication is a promising candidate for future communications due to its flexibility and sufficient bandwidth. However, random fluctuations in the position of hovering UAVs will lead to random variations in the blockage and signal-to-noise ratio (SNR) of the UAV-user link, thus affecting the quality of service (QoS) of the system. To assess the impact of UAV position fluctuations on the QoS of air-to-ground mmWave UAV communications, this paper develops a tractable analytical model that jointly captures the features of three-dimensional (3D) position fluctuations of hovering UAVs and blockages of mmWave (including static, dynamic, and self-blockages). With this model, we derive the closed-form expressions for reliable service probability respective to blockage probability of UAV-user links, and coverage probability respective to SNR, respectively. The results indicate that the greater the position fluctuations of UAVs, the lower the reliable service probability and coverage probability. The degradation of these two evaluation metrics confirms that the performance of air-to-ground mmWave UAV systems largely depends on the UAV position fluctuations, and the stronger the fluctuation, the worse the QoS. Finally, Monte Carlo simulations demonstrate the above results and show UAVs' optimal location to maximize the reliable service and coverage probability, respectively.