Deep learning-based aerial detection is an essential component in modern aircraft, providing fundamental functions such as navigation and situational awareness. Though promising, aerial detection has been shown to be vulnerable to adversarial attacks, posing significant safety concerns. The sparsity of a comprehensive analysis on the robustness of aerial detection exacerbates these vulnerabilities, increasing the risks associated with the practical application of these systems. To bridge this gap, this paper comprehensively studies the potential threats caused by adversarial attacks on aerial detection and analyzes their impact on current defenses. Based on the most widely adopted sensing strategies in aerial detection, we categorize both digital and physical adversarial attacks across optical sensing, infrared sensing, and Synthetic Aperture Radar (SAR) imaging sensing. Owing to the different imaging principles, attacks in each sensing dimension show different attack vectors and reveal varying attack potentials. Additionally, according to the operational life cycles, we analyze adversarial defenses across three operational phases: pre-mission, in-mission, and post-mission. Our findings reveal critical insights into the weaknesses of current systems and offer recommendations for future research directions. This study underscores the importance of addressing the identified challenges in adversarial attack and defense, particularly in real-world scenarios. By focusing future research on enhancing the physical robustness of detection systems, developing comprehensive defense evaluation frameworks, and leveraging high-quality platforms, we can significantly improve the robustness and reliability of aerial detection systems against adversarial threats.