Deep-learning-based object detection is widely used in unmanned aerial vehicle (UAV) systems. However, existing methods can only perform the object detection algorithm on visible images. In many scenarios, the infrared image performs better than the visible image because it can represent more invisible features. This paper proposes an object detection algorithm using image fusion to fully use the advantages of both visible light images and infrared images. Moreover, we optimize and re-design the standard object detection algorithm, YOLO V2, to improve its performance on embedded platforms. By evaluating the performance of the proposed method, we found that the recognition accuracy rate is 97.43%, while the recognition accuracy rate of the visible image is 91.09% and the recognition accuracy rate of the infrared image is 91.39%. The experimental results indicate that the proposed method can increase the recognition accuracy rate of the visible image by 6.34% and increase the recognition accuracy rate of the infrared image by 6.04%.