Citrus fruit detection can provide technical support for fine management and yield determination of citrus orchards. Accurate detection of citrus fruits in mountain orchards is challenging because of leaf occlusion and citrus fruit mutual occlusion of different fruits. This paper presents a citrus detection task that combines UAV data collection, AI embedded device, and target detection algorithm. The system used a small unmanned aerial vehicle equipped with a camera to take full-scale pictures of citrus trees; at the same time, we extended the state-of-the-art model target detection algorithm, added the attention mechanism and adaptive fusion feature method, improved the model’s performance; to facilitate the deployment of the model, we used the pruning method to reduce the amount of model calculation and parameters. The improved target detection algorithm is ported to the edge computing end to detect the data collected by the unmanned aerial vehicle. The experiment was performed on the self-made citrus dataset, the detection accuracy was 93.32%, and the processing speed at the edge computing device was 180 ms/frame. This method is suitable for citrus detection tasks in the mountainous orchard environment, and it can help fruit growers to estimate their yield.