In radiation therapy, for accurate radiation dose delivery to a target tumor and reduction of the extra exposure of normal tissues, real-time tumor tracking is typically an important technique in lung cancer treatment since lung tumors move with patients' respiration. To observe a tumor motion in real time, x-ray fluoroscopic devices can be employed, and various tracking techniques have been proposed to track tumors. However, development of a fast and accurate tracking method for clinical use is still a challenging task since the obscured image of the tumor can cause decreased tracking accuracy and can result in additional processing time for remedying the accuracy. In this study, a new key-point-based tumor tracking method, which is sufficiently fast and accurate, is presented. Given an x-ray image sequence, the proposed method employs a difference-of-Gaussians filtering technique to detect key points in the tumor region of the first frame which are robust against noise and outliers in the subsequent frames. In the subsequent frames, these key points are tracked using a fast optical flow technique, and tumor motion is estimated via their movement. To evaluate the performance, the proposed method has been tested on several clinical kV and MV x-ray image sequences. The experimental results showed that the average of the root mean square errors of tracking were [Formula: see text] and [Formula: see text] for kV and MV x-ray image sequences, respectively. This tracking performance was more accurate than previous tracking methods. In addition, the average processing times for each frame were [Formula: see text] and [Formula: see text] for kV and MV image sequences, respectively, and the proposed method was faster than previous methods as well as shorter than frame acquisition interval. Therefore, the proposed method has the potential for both highly accurate and fast tumor tracking in clinical applications.