Vision-based object tracking is crucial for both civil and military applications. A range of hazards to cyber safety, vital infrastructure, and public privacy are posed by the rise of drones, or unmanned aerial vehicles (UAV). As a result, identifying suspicious drones/UAV is a serious issue that has attracted attention recently. The key focus of this research is to develop a unique virtual coloured marker based tracking algorithm to recognise and predict the pose of a detected object within the camera field-of-view. After detecting the object, proposed method begins by determining the area of detected object as reference-contour. Following that, a Virtual-Bounding Box (V-BB) is developed over the reference-contour by meeting the minimum area of contour criteria. In order to track and estimate the precise location of the detected object in two-dimensions during observations, a Virtual Dynamic Crossline with a Virtual Static Graph (VDC-VSG) was constructed to follow the motion of V-BB, which is considered as a virtual coloured marker. Additionally, the virtual coloured marker helps to avoid issues linked to ambient lighting and chromatic variation. To some extent, it can function efficiently during obstructions like rapid position fluctuations, low resolution and noises etc. The efficacy of the developed algorithm is evaluated by testing with significant number of aerial sequences, including benchmark footage and the outputs were outstanding, with better results. The suggested method will support future industry of computer vision-based intelligent systems. Potential applications of the proposed method includes object detection and analysis applied to the field of security and defence.