The widespread growth of drone technology is generating new security paradigms, especially with regard to the unauthorized activities of UAVs in restricted or sensitive areas, as well as illegal and illicit activities or attacks. Among the various UAV detection technologies, vision systems in different spectra are postulated as outstanding technologies due to their peculiarities compared to other technologies. However, drone detection in thermal imaging is a challenging task due to specific factors such as thermal noise, temperature variability, or cluttered environments. This study addresses these challenges through a comparative evaluation of contemporary neural network architectures—specifically, convolutional neural networks (CNNs) and transformer-based models—for UAV detection in infrared imagery. The research focuses on real-world conditions and examines the performance of YOLOv9, GELAN, DETR, and ViTDet in different scenarios of the Anti-UAV Challenge 2023 dataset. The results show that YOLOv9 stands out for its real-time detection speed, while GELAN provides the highest accuracy in varying conditions and DETR performs reliably in thermally complex environments. The study contributes to the advancement of state-of-the-art UAV detection techniques and highlights the need for the further development of specialized models for specific detection scenarios.