Purpose: Determine the time-dependent magnitude of intrafraction prostate displacement and a cutoff for the tracking decision. Methods: Nine patients with localized prostate cancer were treated with ultra-hypofractionated radiotherapy (CyberKnife) with fiducial markers. Exact tract kV/kV imaging was used with an average interval of 19–92 s. A Gaussian distribution was calculated for the x-, y-, and z-directions (σx,y,z). The variation of prostate motion (μσ) was obtained by averaging the patients’ specifics, and the safety margin was calculated to be MAB = WYm + WBSs. Results: The calculated PTV safety margins were as follows: at 40 s: 0.55 mm (L/r), 0.85 mm (a/p), and 1.05 mm (s/i); at 60 s: 0.9 mm (L/r), 1.35 mm (a/p), and 1.55 mm (s/i); at 100 s: 1.5 mm (L/r), 2.3 mm (a/p), and 2.6 mm (s/i); at 150 s: 1.9 mm (L/r), 3.1 mm (a/p), and 3.6 mm (s/i); at 200 s: 2.2 mm (L/r), 3.8 mm (a/p), and 4.2 mm (s/i); and at 300 s: 2.6 mm (L/r), 5.3 mm (a/p), and 5.6 mm (s/i). A tracking cutoff of 2.5 min seemed reasonable. In order to achieve an accuracy of < 1 mm, tracking with < 50 s intervals was necessary. Conclusions: For ultra-hypofractionated radiotherapy of the prostate with treatment times > 2.5 min, intrafraction motion management is recommended.