In computer assisted orthopaedic surgery, it is important to find the correct spatial lo- cation of the target in a predefined world coordinate, so that the model can be transformed accordingly onto the surgical site for surgeons’ reference. Current tracking systems mainly rely on the detection of optical markers inserted into the anatomy. The invasiveness of fixa- tion pins increases operating time and bone complications. Automatic markerless tracking is therefore preferred in practice. In this paper, we integrate an automatic RGBD-image based segmentation neural network and a fast markerless registration algorithm to achieve the markerless tracking purpose. An experimental test with a metal leg was designed. By forcing the alignment of the measured hip joint centre, the overall tracking was shown to be sub-degree in terms of orientation accuracy, which is clinically acceptable.