This paper presents the implementation of a multi-7 threaded parallel architecture, which enables telescope-based op-8 tical UAV detection and tracking in real-time. For efficient image 9 processing an accurate deep learning object detector is comple-10 mented in parallel by a fast object tracker. A transition strategy 11 between detector and tracker is introduced based on the tracker 12 reliability, which improves the object localization accuracy of 13 the system. The deep learning algorithm initializes the tracker 14 and in the subsequent frames the reliability of the tracker is 15 compared to the confidence value of each newly detected object to 16 determine whether a reinitialization is necessary. The implemented 17 architecture successfully demonstrates the parallel combination of 18 an FRCNN detector and a MEDIANFLOW tracker to achieve visual 19 UAV detection and tracking at 100 fps. The proposed reliability-20 based strategy outperforms a purely detector and tracker-based 21 strategy by 6% and 14% respectively in terms of intersection 22 over union at a threshold of 0.5, in scenarios, when the target 23 UAV is flying in front of a complex background. Additionally, the 24 implemented parallel architecture increases the probability for a 25 flight path estimation, which requires at least two localizations, by 26 49%, when compared to a non-parallel architecture. Field tests are 27 conducted with the proposed architecture using a telescope system 28 demonstrating UAV detection and tracking at 100 fps in distances 29 up to 4000 m in front of a clear background.30