Visual tracking for augmented reality tours is still challenging for cultural heritage sites because of the great variation of tracking targets and environments on such sites. Even at today's state of the art, it is almost impossible to apply just one tracking method to all the various environments with any hope of success. This paper presents a tracking framework to overcome this problem. It consists of different tracking flows, each efficiently using robust visual cues of the target scene. Analysis of the tracking environment enables more practical tracking at the sites. The reliability of the tracking framework is verified through on-site demonstrations at Gyeongbokgung, the most symbolic cultural heritage site in Korea.