Purpose: To assess the optimal planning target volume (PTV) margins for stereotactic body radiotherapy (SBRT) of prostate cancer based on inter-and intra-fractional prostate motion determined from daily image guidance. Methods and Materials: Two hundred and five patients who were enrolled on two prospective studies of SBRT (8 Gy × 5 fractions) for localized prostate cancer treated at a single institution between 2012 and 2017 had complete inter-and intra-fractional shift data available. All patients had scheduled kilovoltage planar imaging during SBRT with rigid registration to intraprostatic fiducials prior to each of four half-arcs delivered per fraction, as well as cone beam CT verification of anatomy prior to each fraction. Inter-and intra-fractional shift data were obtained to estimate the required PTV margins based on the classic van Herk formula. Inter-and intra-fractional motion were compared between patients with and without severe toxicities using the independent two-sample Wilcoxon test. Results: The margins required to account for inter-fractional motion were estimated to be 0.99, 1.52, and 1.45 cm in lateral (LR), longitudinal (SI), and vertical (AP) directions, respectively. The margins required to account for intra-fractional motion were estimated to be 0.19, 0.27, and 0.31 cm in LR, SI and AP directions, respectively. Large intra-fractional shifts were mostly observed in the SI and AP directions, with 2.0 and 5.4% of patients experiencing average intra-fractional motion >3 mm in the SI and AP directions, respectively, compared with none experiencing mean shifts >3 mm in the LR direction. Six patients experienced grade 3 gastrointestinal or genitourinary toxicity. There were no significant differences in mean inter-or intra-fractional motion in any of the cardinal directions compared to patients without severe toxicity (inter-fractional p = 0.46-0.99, intra-fractional p = 0.10-0.84). Levin-Epstein et al. Prostate Displacement and SBRT Margins Conclusion: The inter-and intra-fractional margins estimated from this study are in line with prior reported values. Intra-fractional prostate motion was generally small with larger margins required for the SI and AP directions, notably just slightly exceeding the commonly used 3 mm posterior PTV margin even with realignment between half-arcs. Development of severe toxicity was not significantly associated with the degree of inter-or intra-fractional motion.