Background: The cornerstone of management of phyllodes tumor is surgery. No standard of care exists regarding adjuvant therapy; however, local recurrence is a predominant pattern of failure. The aims of this study were to evaluate the clinical characteristics and treatment patterns of phyllodes tumor and to compare local recurrence and disease-free survival rates of patients with borderline and malignant phyllodes tumor treated with or without adjuvant radiotherapy. Materials and Methods: We analyzed the demographic data, treatment details, and recurrence patterns of all patients with nonmetastatic phyllodes tumors of the breast (n = 34) treated with a multimodality approach who presented to our institute from January 2015 to December 2020. Results: The median age at presentation was 41.2 years, and the median tumor size was 12.5 cm. All patients underwent definitive surgical procedures in the form of wide local excision or mastectomy. No recurrence was noted in the patients with a benign histology. After a median follow-up period of 38 months, the local recurrence rate was 27% for the patients with borderline and malignant histology treated with adjuvant radiotherapy (n = 11) versus 47% for those (n = 17) who did not receive adjuvant radiotherapy. The 3-year local recurrence-free survival rate was 72% in the adjuvant radiotherapy group, versus 51% in the surgery only group. Conclusion: The results of the current study confirm the excellent prognosis of patients with benign phyllodes tumor undergoing surgery alone. Local recurrence was the predominant mode of failure in the patients with borderline and malignant histology. Locoregional control was improved with the addition of postoperative radiotherapy in the patients with borderline and malignant histology irrespective of margin status.