This paper proposes an image processing algorithm for detection and tracking of aerial vehicles in sight. The proposed algorithm detects moving objects using the image homography calculated from a video stream taken from the onboard camera and determines whether the detected objects are approaching aerial vehicles by the Probabilistic Multi-Hypothesis Tracking (PMHT) method. This algorithm performs well especially when it is needed to detect any approaching aircraft seen with cluttered background. Further, our algorithm is suitable for real flight application as it is less sensitive to light conditions or color variations. The performance of the proposed algorithm is validated by applying it to the onboard video clips taken during actual flights using two unmanned aerial vehicles.