The ability to monitor tumor motion without implanted markers can potentially enable broad access to more accurate and precise lung radiotherapy. A major challenge is that kilovoltage (kV) imaging based methods are rarely able to continuously track the tumor due to the inferior tumor visibility on 2D kV images. Another challenge is the estimation of 3D tumor position based on only 2D imaging information. The aim of this work is to address both challenges by proposing a Bayesian approach for markerless tumor tracking for the first time. The proposed approach adopts the framework of the extended Kalman filter, which combines a prediction and measurement steps to make the optimal tumor position update. For each imaging frame, the tumor position is first predicted by a respiratory-correlated model. The 2D tumor position on the kV image is then measured by template matching. Finally, the prediction and 2D measurement is combined based on the 3D distribution of tumor positions in the past 10 seconds and the estimated uncertainty of template matching. To investigate the clinical feasibility of the proposed method, a total of 13 lung cancer patient datasets were used for retrospective validation, including 11 cone-beam CT scan pairs and two stereotactic ablative body radiotherapy cases. The ground truths for tumor motion were generated from the the 3D trajectories of implanted markers or beacons. The mean, standard deviation, and 95th percentile of the 3D tracking error was found to range from 1.6–2.9 mm, 0.6–1.5 mm, and 2.6–5.8 mm, respectively. Markerless tumor tracking always resulted in smaller errors compared to the standard of care. The improvement was the most pronounced in the superior-inferior (SI) direction, with up to 9.5 mm reduction in the 95th-percentile SI error for patients with >10 mm 5th-to-95th percentile SI tumor motion. The percentage of errors with 3D magnitude <5 mm was 96.5% for markerless tumor tracking and 84.1% for the standard of care. The feasibility of markerless tumor tracking has been demonstrated on realistic clinical scenarios for the first time. The clinical implementation of the proposed method will enable more accurate and precise lung radiotherapy using existing hardware and workflow. Future work is focused on the clinical and real-time implementation of this method.