The widespread use of unmanned aerial vehicles (UAVs) has brought many benefits, particularly for military and civil applications. For example, UAVs can be used in communication, ecological surveys, agriculture, and logistics to improve efficiency and reduce the required workforce. However, the malicious use of UAVs can significantly endanger public safety and pose many challenges to society. Therefore, detecting malicious UAVs is an important and urgent issue that needs to be addressed. In this study, a combined UAV detection model (CUDM) based on analyzing video abnormal behavior is proposed. CUDM uses abnormal behavior detection models to improve the traditional object detection process. The work of CUDM can be divided into two stages. In the first stage, our model cuts the video into images and uses the abnormal behavior detection model to remove a large number of useless images, improving the efficiency and real-time detection of suspicious targets. In the second stage, CUDM works to identify whether the suspicious target is a UAV or not. Besides, CUDM relies only on ordinary equipment such as surveillance cameras, avoiding the use of expensive equipment such as radars. A self-made UAV dataset was constructed to verify the reliability of CUDM. The results show that CUDM not only maintains the same accuracy as state-of-the-art object detection models but also reduces the workload by 32%. Moreover, it can detect malicious UAVs in real-time.