This paper presents methods and algorithms for real-time visual target detection, recognition and tracking, both in the case of ground-based objects (surveyed from a moving airborne imaging sensor) and flying targets (observed from a groundbased or vehicle mounted sensor). The methods are highly parallelized and partially implemented on GPU, with the goal of real-time speeds even in the case of multiple target observations. Real-time applicability is in focus. The methods use single camera observations, providing a passive and expendable alternative for expensive and/or active sensors. Use cases involve perimeter defense and surveillance situations, where passive detection and observation is a priority (e.g. aerial surveillance of a compound, detection of reconnaissance drones, etc.).